Building

Started building at 2021/12/11 09:14:03
Calculating base
Updating mirror
Basing run on 7.1.0-1893 20d83986c
Updating tree for run 11.12.2021-09.14
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 888c51e, changes since last good build: none
kv_engine is at 2499d82, changes since last good build: none
ns_server is at 5bd87dc, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 2f97349, changes since last good build: 
 2f9734919 add prepared query context tests plus a few other query context tests.
 94db07486 Adding support for 6.6.5 and 7.0.3 in testrunner
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable e94da62a] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable af652aba] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable c7cfa2bb] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable d9949333] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 0cba7ab7] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 408af667] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable dbef4005] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable 20a1ae7b] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable b91580a] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/11 09:30:10
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/11 09:31:42 Started functional
2021/12/11 09:31:50 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/11 09:31:56 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/11 09:31:56 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 09:32:02 Removing node(s): [] from the cluster
2021/12/11 09:32:07 Rebalance progress: 0
2021/12/11 09:32:12 Rebalance progress: 0.3616898148148148
2021/12/11 09:32:17 Rebalance progress: 5.081741898148148
2021/12/11 09:32:22 Rebalance progress: 9.25925925925926
2021/12/11 09:32:27 Rebalance progress: 13.2740162037037
2021/12/11 09:32:32 Rebalance progress: 18.03023726851852
2021/12/11 09:32:37 Rebalance progress: 21.32161458333333
2021/12/11 09:32:42 Rebalance progress: 25.95124421296296
2021/12/11 09:32:47 Rebalance progress: 100
2021/12/11 09:32:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:33:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:33:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:33:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:33:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:34:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:34:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:34:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:34:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:35:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:35:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:35:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:35:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:36:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:36:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:36:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:36:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:37:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:37:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:37:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/11 09:37:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/11 09:38:08 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:38:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:38:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 09:38:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 09:38:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 09:38:29 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/11 09:38:29 App: TestAdvancedGetOps got deployed
2021/12/11 09:38:29 Starting bucket ops to source bucket
2021/12/11 09:38:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 09:38:39 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 09:38:39 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 09:38:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:38:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:38:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:38:44 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 09:38:49 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 09:38:54 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 09:38:59 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/11 09:38:59 App: TestAdvancedGetOps got un-deployed
2021/12/11 09:38:59 No eventing-con process running
2021/12/11 09:38:59 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/11 09:39:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:39:11 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/11 09:39:16 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/11 09:39:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:39:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:39:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:39:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (80.08s)
=== RUN   TestAdvancedInsertOps
2021/12/11 09:39:28 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:39:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:39:34 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:39:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:39:44 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:39:49 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:39:49 App: TestAdvancedInsertOps got deployed
2021/12/11 09:39:49 Starting bucket ops to source bucket
2021/12/11 09:39:55 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 09:40:00 Waiting for dst bucket item count to get to: 1024 curr count: 307
2021/12/11 09:40:05 Waiting for dst bucket item count to get to: 1024 curr count: 819
2021/12/11 09:40:05 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/11 09:40:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:40:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:40:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:40:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:40:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:40:20 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:40:25 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:40:25 App: TestAdvancedInsertOps got un-deployed
2021/12/11 09:40:25 No eventing-con process running
2021/12/11 09:40:25 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/11 09:40:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:40:37 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 09:40:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:40:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:40:48 Waiting for dst bucket item count to get to: 0 curr count: 828
2021/12/11 09:40:53 Waiting for dst bucket item count to get to: 0 curr count: 316
2021/12/11 09:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:40:53 Testing insert operation on source bucket
2021/12/11 09:40:53 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:40:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:40:59 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:41:04 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:41:09 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:41:14 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/11 09:41:14 App: TestAdvancedInsertOps got deployed
2021/12/11 09:41:14 Starting bucket ops to source bucket
2021/12/11 09:41:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/11 09:41:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/11 09:41:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/11 09:41:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:41:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:41:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:41:31 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:41:36 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:41:41 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:41:46 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/11 09:41:46 App: TestAdvancedInsertOps got un-deployed
2021/12/11 09:41:46 No eventing-con process running
2021/12/11 09:41:46 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/11 09:41:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:41:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/11 09:41:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:41:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:42:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.43s)
=== RUN   TestAdvancedUpsertOps
2021/12/11 09:42:04 Starting bucket ops to source bucket
2021/12/11 09:42:05 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:42:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:42:11 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:42:16 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:42:21 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:42:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:42:26 App: TestAdvancedUpsertOps got deployed
2021/12/11 09:42:26 Starting bucket ops to source bucket
2021/12/11 09:42:32 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 09:42:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:42:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:42:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:42:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:42:37 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:42:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:42:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:42:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:42:52 App: TestAdvancedUpsertOps got un-deployed
2021/12/11 09:42:52 No eventing-con process running
2021/12/11 09:42:52 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/11 09:42:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:43:04 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 09:43:09 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/11 09:43:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:43:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:43:15 Testing upsert operation on source bucket
2021/12/11 09:43:15 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:43:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:43:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:43:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:43:32 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:43:37 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/11 09:43:37 App: TestAdvancedUpsertOps got deployed
2021/12/11 09:43:37 Starting bucket ops to source bucket
2021/12/11 09:43:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:43:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:43:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:43:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:43:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:43:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:43:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:43:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/11 09:43:58 App: TestAdvancedUpsertOps got un-deployed
2021/12/11 09:43:58 No eventing-con process running
2021/12/11 09:43:58 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/11 09:43:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:44:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:44:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (127.67s)
=== RUN   TestAdvancedReplaceOps
2021/12/11 09:44:12 Starting bucket ops to source bucket
2021/12/11 09:44:13 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:44:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:44:19 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:44:24 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:44:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:44:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:44:34 App: TestAdvancedReplaceOps got deployed
2021/12/11 09:44:34 Starting bucket ops to source bucket
2021/12/11 09:44:40 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 09:44:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 09:44:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:44:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:44:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:44:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:44:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:44:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:45:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:45:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:45:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/11 09:45:05 No eventing-con process running
2021/12/11 09:45:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/11 09:45:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:45:17 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 09:45:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:45:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:45:23 Testing upsert operation on source bucket
2021/12/11 09:45:23 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:45:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:45:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:45:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:45:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:45:45 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/11 09:45:45 App: TestAdvancedReplaceOps got deployed
2021/12/11 09:45:45 Starting bucket ops to source bucket
2021/12/11 09:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:45:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:45:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:45:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:45:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:45:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:46:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:46:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/11 09:46:06 App: TestAdvancedReplaceOps got un-deployed
2021/12/11 09:46:06 No eventing-con process running
2021/12/11 09:46:06 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/11 09:46:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:46:18 Waiting for dst bucket item count to get to: 0 curr count: 125
2021/12/11 09:46:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:46:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:46:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.68s)
=== RUN   TestAdvancedDeleteOps
2021/12/11 09:46:24 Starting bucket ops to source bucket
2021/12/11 09:46:24 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:46:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:46:31 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:46:36 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:46:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:46:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:46:46 App: TestAdvancedDeleteOps got deployed
2021/12/11 09:46:46 Starting bucket ops to source bucket
2021/12/11 09:46:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:46:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:46:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:46:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:46:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:46:56 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:47:01 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:47:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:47:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:47:11 App: TestAdvancedDeleteOps got un-deployed
2021/12/11 09:47:11 No eventing-con process running
2021/12/11 09:47:11 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/11 09:47:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:47:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:47:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:47:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:47:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:47:34 Testing delete operation on source bucket
2021/12/11 09:47:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:47:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/11 09:47:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:47:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:47:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:47:56 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/11 09:47:56 App: TestAdvancedDeleteOps got deployed
2021/12/11 09:47:56 Starting bucket ops to source bucket
2021/12/11 09:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:47:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:47:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:47:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:48:01 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:48:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:48:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:48:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/11 09:48:16 App: TestAdvancedDeleteOps got un-deployed
2021/12/11 09:48:16 No eventing-con process running
2021/12/11 09:48:16 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/11 09:48:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:48:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:48:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:48:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.60s)
=== RUN   TestEnoentAdvancedGet
2021/12/11 09:48:30 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:48:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/11 09:48:36 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:48:41 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:48:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:48:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:48:51 App: TestEnoentAdvancedGet got deployed
2021/12/11 09:48:51 Starting bucket ops to source bucket
2021/12/11 09:48:56 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 09:49:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 09:49:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 09:49:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:49:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:49:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:49:06 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:49:11 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:49:16 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:49:21 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:49:21 App: TestEnoentAdvancedGet got un-deployed
2021/12/11 09:49:21 No eventing-con process running
2021/12/11 09:49:21 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/11 09:49:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:49:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:49:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:49:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:49:50 Waiting for dst bucket item count to get to: 0 curr count: 57
2021/12/11 09:49:55 Waiting for dst bucket item count to get to: 0 curr count: 57
2021/12/11 09:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:49:55 Testing get enoent operation on source bucket
2021/12/11 09:49:55 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:49:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/11 09:50:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:50:06 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:50:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:50:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/11 09:50:16 App: TestEnoentAdvancedGet got deployed
2021/12/11 09:50:16 Starting bucket ops to source bucket
2021/12/11 09:50:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/11 09:50:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/11 09:50:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:50:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:50:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:50:26 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:50:31 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:50:36 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:50:41 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/11 09:50:41 App: TestEnoentAdvancedGet got un-deployed
2021/12/11 09:50:41 No eventing-con process running
2021/12/11 09:50:41 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/11 09:50:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:50:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 09:50:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:50:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.99s)
=== RUN   TestEnoentAdvancedReplace
2021/12/11 09:51:05 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:51:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 09:51:11 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:51:16 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:51:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:51:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:51:26 App: TestEnoentAdvancedReplace got deployed
2021/12/11 09:51:26 Starting bucket ops to source bucket
2021/12/11 09:51:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 09:51:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 09:51:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:51:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:51:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:51:36 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:51:41 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:51:46 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:51:51 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:51:51 App: TestEnoentAdvancedReplace got un-deployed
2021/12/11 09:51:51 No eventing-con process running
2021/12/11 09:51:51 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/11 09:51:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:52:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:52:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:52:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:52:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:52:20 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/11 09:52:25 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/11 09:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:52:25 Testing get enoent operation on source bucket
2021/12/11 09:52:25 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:52:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 09:52:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:52:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:52:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:52:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/11 09:52:46 App: TestEnoentAdvancedReplace got deployed
2021/12/11 09:52:46 Starting bucket ops to source bucket
2021/12/11 09:52:51 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/11 09:52:51 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/11 09:52:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:52:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:52:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:52:56 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:53:01 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:53:06 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:53:11 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/11 09:53:11 App: TestEnoentAdvancedReplace got un-deployed
2021/12/11 09:53:11 No eventing-con process running
2021/12/11 09:53:11 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/11 09:53:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:53:23 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 09:53:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:53:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.90s)
=== RUN   TestEnoentAdvancedDelete
2021/12/11 09:53:35 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:53:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 09:53:41 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:53:46 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:53:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:53:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:53:56 App: TestEnoentAdvancedDelete got deployed
2021/12/11 09:53:56 Starting bucket ops to source bucket
2021/12/11 09:54:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 09:54:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 09:54:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:54:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:54:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:54:06 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:54:11 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:54:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:54:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:54:21 App: TestEnoentAdvancedDelete got un-deployed
2021/12/11 09:54:21 No eventing-con process running
2021/12/11 09:54:21 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/11 09:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:54:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:54:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:54:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:54:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:54:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:54:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:54:55 Testing get enoent operation on source bucket
2021/12/11 09:54:55 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:54:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 09:55:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:55:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:55:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:55:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/11 09:55:16 App: TestEnoentAdvancedDelete got deployed
2021/12/11 09:55:16 Starting bucket ops to source bucket
2021/12/11 09:55:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/11 09:55:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/11 09:55:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:55:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:55:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:55:26 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:55:31 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:55:36 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:55:41 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/11 09:55:41 App: TestEnoentAdvancedDelete got un-deployed
2021/12/11 09:55:41 No eventing-con process running
2021/12/11 09:55:41 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/11 09:55:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:55:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 09:55:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:55:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:56:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.89s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/11 09:56:05 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:56:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/11 09:56:11 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:56:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:56:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:56:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:56:26 App: TestAdvancedInsertKeyExist got deployed
2021/12/11 09:56:26 Starting bucket ops to source bucket
2021/12/11 09:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:56:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:56:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:56:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:56:31 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:56:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:56:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:56:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:56:46 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/11 09:56:46 No eventing-con process running
2021/12/11 09:56:46 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/11 09:56:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:56:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:57:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:57:00 Pumping document in the destination bucket
2021/12/11 09:57:00 Starting bucket ops to source bucket
2021/12/11 09:57:01 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:57:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/11 09:57:07 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:57:12 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:57:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:57:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/11 09:57:22 App: TestAdvancedInsertKeyExist got deployed
2021/12/11 09:57:22 Starting bucket ops to source bucket
2021/12/11 09:57:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:57:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:57:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:57:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:57:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:57:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:57:37 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:57:42 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:57:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:57:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/11 09:57:52 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/11 09:57:52 No eventing-con process running
2021/12/11 09:57:52 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/11 09:57:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:58:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:58:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 09:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:58:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:58:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.67s)
=== RUN   TestExipryGet
2021/12/11 09:58:15 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:58:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/11 09:58:22 Waiting for app: TestExipryGet to get deployed
2021/12/11 09:58:27 Waiting for app: TestExipryGet to get deployed
2021/12/11 09:58:32 Waiting for app: TestExipryGet to get deployed
2021/12/11 09:58:37 Waiting for app: TestExipryGet to get deployed
2021/12/11 09:58:37 App: TestExipryGet got deployed
2021/12/11 09:58:37 Starting bucket ops to source bucket
2021/12/11 09:59:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 09:59:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:59:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 09:59:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 09:59:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 09:59:27 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 09:59:32 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 09:59:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 09:59:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 09:59:42 App: TestExipryGet got un-deployed
2021/12/11 09:59:42 No eventing-con process running
2021/12/11 09:59:42 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/11 09:59:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:59:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 09:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 09:59:56 Pumping document in the destination bucket for expiry get
2021/12/11 09:59:56 Starting bucket ops to source bucket
2021/12/11 09:59:56 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 09:59:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/11 10:00:02 Waiting for app: TestExipryGet to get deployed
2021/12/11 10:00:07 Waiting for app: TestExipryGet to get deployed
2021/12/11 10:00:12 Waiting for app: TestExipryGet to get deployed
2021/12/11 10:00:17 Waiting for app: TestExipryGet to get deployed
2021/12/11 10:00:17 App: TestExipryGet got deployed
2021/12/11 10:00:17 Starting bucket ops to source bucket
2021/12/11 10:01:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:01:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:01:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:01:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:01:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:01:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:01:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:01:18 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 10:01:23 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 10:01:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 10:01:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/11 10:01:33 App: TestExipryGet got un-deployed
2021/12/11 10:01:33 No eventing-con process running
2021/12/11 10:01:33 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/11 10:01:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:01:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:01:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:01:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:01:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (215.80s)
=== RUN   TestExipryInsert
2021/12/11 10:01:51 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:01:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/11 10:01:58 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:02:03 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:02:08 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:02:13 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:02:13 App: TestExipryInsert got deployed
2021/12/11 10:02:13 Starting bucket ops to source bucket
2021/12/11 10:02:58 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 10:03:03 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/11 10:03:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:03:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:03:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:03:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:03:08 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:03:13 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:03:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:03:23 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:03:23 App: TestExipryInsert got un-deployed
2021/12/11 10:03:23 No eventing-con process running
2021/12/11 10:03:23 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/11 10:03:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:03:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:03:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:03:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:03:37 Testing on the destination bucket for expiry insert
2021/12/11 10:03:37 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:03:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/11 10:03:43 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:03:48 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:03:53 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:03:58 Waiting for app: TestExipryInsert to get deployed
2021/12/11 10:03:58 App: TestExipryInsert got deployed
2021/12/11 10:03:58 Starting bucket ops to source bucket
2021/12/11 10:04:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:04:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:04:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:04:48 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:04:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:04:53 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:04:58 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:05:03 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:05:08 Waiting for app: TestExipryInsert to get un-deployed
2021/12/11 10:05:08 App: TestExipryInsert got un-deployed
2021/12/11 10:05:08 No eventing-con process running
2021/12/11 10:05:08 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/11 10:05:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:05:20 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/11 10:05:25 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/11 10:05:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:05:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:05:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.59s)
=== RUN   TestExipryReplace
2021/12/11 10:05:32 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:05:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 10:05:38 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:05:43 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:05:48 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:05:53 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:05:53 App: TestExipryReplace got deployed
2021/12/11 10:05:53 Starting bucket ops to source bucket
2021/12/11 10:06:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:06:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:06:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:06:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:06:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:06:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:06:48 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:06:53 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:06:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:07:03 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:07:03 App: TestExipryReplace got un-deployed
2021/12/11 10:07:03 No eventing-con process running
2021/12/11 10:07:03 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/11 10:07:05 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:07:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:07:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:07:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:07:17 Testing on the destination bucket for expiry insert
2021/12/11 10:07:17 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:07:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 10:07:23 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:07:28 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:07:34 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:07:39 Waiting for app: TestExipryReplace to get deployed
2021/12/11 10:07:39 App: TestExipryReplace got deployed
2021/12/11 10:07:39 Starting bucket ops to source bucket
2021/12/11 10:08:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:08:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:08:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:08:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:08:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:08:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:08:34 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:08:39 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:08:44 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:08:49 Waiting for app: TestExipryReplace to get un-deployed
2021/12/11 10:08:49 App: TestExipryReplace got un-deployed
2021/12/11 10:08:49 No eventing-con process running
2021/12/11 10:08:49 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/11 10:08:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:09:01 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/11 10:09:06 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/11 10:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:09:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:09:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.50s)
=== RUN   TestExipryUpsert
2021/12/11 10:09:12 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:09:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/11 10:09:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:09:24 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:09:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:09:34 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:09:34 App: TestExipryUpsert got deployed
2021/12/11 10:09:34 Starting bucket ops to source bucket
2021/12/11 10:10:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:10:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:10:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:10:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:10:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:10:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:10:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:10:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:10:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:10:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:10:44 App: TestExipryUpsert got un-deployed
2021/12/11 10:10:44 No eventing-con process running
2021/12/11 10:10:44 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/11 10:10:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:10:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:10:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:10:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:10:58 Testing on the destination bucket for expiry insert
2021/12/11 10:10:58 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:10:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/11 10:11:04 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:11:09 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:11:14 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:11:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/11 10:11:19 App: TestExipryUpsert got deployed
2021/12/11 10:11:19 Starting bucket ops to source bucket
2021/12/11 10:12:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:12:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:12:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:12:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:12:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:12:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:12:14 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:12:19 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:12:24 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:12:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/11 10:12:29 App: TestExipryUpsert got un-deployed
2021/12/11 10:12:29 No eventing-con process running
2021/12/11 10:12:29 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/11 10:12:31 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:12:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:12:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:12:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:12:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:12:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.60s)
=== RUN   TestCasReplace
2021/12/11 10:12:53 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:12:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 10:12:59 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:13:04 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:13:09 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:13:14 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:13:14 App: TestCasReplace got deployed
2021/12/11 10:13:14 Starting bucket ops to source bucket
2021/12/11 10:13:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:13:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:13:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:13:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:13:19 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:13:24 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:13:29 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:13:34 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:13:34 App: TestCasReplace got un-deployed
2021/12/11 10:13:34 No eventing-con process running
2021/12/11 10:13:34 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/11 10:13:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:13:46 Waiting for dst bucket item count to get to: 0 curr count: 9
2021/12/11 10:13:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:13:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:13:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:13:53 Testing on the destination bucket for upsert with cas
2021/12/11 10:13:53 Starting bucket ops to source bucket
2021/12/11 10:13:53 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:13:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/11 10:13:59 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:14:04 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:14:09 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:14:14 Waiting for app: TestCasReplace to get deployed
2021/12/11 10:14:14 App: TestCasReplace got deployed
2021/12/11 10:14:14 Starting bucket ops to source bucket
2021/12/11 10:14:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:14:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:14:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:14:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:14:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:14:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:14:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:14:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:14:40 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:14:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/11 10:14:45 App: TestCasReplace got un-deployed
2021/12/11 10:14:45 No eventing-con process running
2021/12/11 10:14:45 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/11 10:14:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:14:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:14:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.15s)
=== RUN   TestCasDelete
2021/12/11 10:15:03 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:15:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 10:15:09 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:15:14 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:15:19 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:15:24 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:15:24 App: TestCasDelete got deployed
2021/12/11 10:15:24 Starting bucket ops to source bucket
2021/12/11 10:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:15:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:15:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:15:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:15:30 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:15:35 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:15:40 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:15:45 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:15:45 App: TestCasDelete got un-deployed
2021/12/11 10:15:45 No eventing-con process running
2021/12/11 10:15:45 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/11 10:15:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:15:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 10:15:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:15:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:16:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:16:03 Testing on the destination bucket for delete with cas
2021/12/11 10:16:03 Starting bucket ops to source bucket
2021/12/11 10:16:03 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:16:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 10:16:09 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:16:14 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:16:19 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:16:24 Waiting for app: TestCasDelete to get deployed
2021/12/11 10:16:24 App: TestCasDelete got deployed
2021/12/11 10:16:24 Starting bucket ops to source bucket
2021/12/11 10:16:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:16:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:16:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:16:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 10:16:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:16:40 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:16:45 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:16:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:16:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/11 10:16:55 App: TestCasDelete got un-deployed
2021/12/11 10:16:55 No eventing-con process running
2021/12/11 10:16:55 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/11 10:16:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:17:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:17:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.18s)
=== RUN   TestCountersIncrement
2021/12/11 10:17:13 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 10:17:19 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 10:17:19 Removing node(s): [] from the cluster
2021/12/11 10:17:24 Rebalance progress: 100
2021/12/11 10:17:24 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:17:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/11 10:17:31 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:17:36 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:17:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:17:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:17:46 App: TestCountersIncrement got deployed
2021/12/11 10:17:46 Starting bucket ops to source bucket
2021/12/11 10:17:52 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 10:17:57 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/11 10:17:57 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/11 10:17:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:17:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:18:02 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:18:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:18:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:18:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:18:17 App: TestCountersIncrement got un-deployed
2021/12/11 10:18:17 No eventing-con process running
2021/12/11 10:18:17 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/11 10:18:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:18:29 Waiting for dst bucket item count to get to: 0 curr count: 966
2021/12/11 10:18:34 Waiting for dst bucket item count to get to: 0 curr count: 453
2021/12/11 10:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:18:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:18:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/11 10:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:18:45 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:18:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/11 10:18:52 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:18:57 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:19:02 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:19:07 Waiting for app: TestCountersIncrement to get deployed
2021/12/11 10:19:07 App: TestCountersIncrement got deployed
2021/12/11 10:19:07 Starting bucket ops to source bucket
2021/12/11 10:19:13 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/11 10:19:18 Waiting for dst bucket item count to get to: 2048 curr count: 1379
2021/12/11 10:19:23 Waiting for dst bucket item count to get to: 2048 curr count: 1379
2021/12/11 10:19:23 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/11 10:19:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:19:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:19:28 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:19:33 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:19:38 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:19:43 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/11 10:19:43 App: TestCountersIncrement got un-deployed
2021/12/11 10:19:43 No eventing-con process running
2021/12/11 10:19:43 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/11 10:19:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:19:55 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/11 10:19:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:19:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:20:07 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 10:20:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:20:07 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/11 10:20:12 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.41s)
=== RUN   TestCountersDecrement
2021/12/11 10:20:12 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 10:20:12 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 10:20:12 Removing node(s): [] from the cluster
2021/12/11 10:20:17 Rebalance progress: 100
2021/12/11 10:20:17 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:20:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/11 10:20:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:20:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:20:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:20:38 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:20:38 App: TestCountersDecrement got deployed
2021/12/11 10:20:38 Starting bucket ops to source bucket
2021/12/11 10:20:45 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/11 10:20:50 Waiting for dst bucket item count to get to: 1024 curr count: 328
2021/12/11 10:20:55 Waiting for dst bucket item count to get to: 1024 curr count: 328
2021/12/11 10:20:55 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/11 10:20:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:20:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:21:00 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:21:05 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:21:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:21:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:21:15 App: TestCountersDecrement got un-deployed
2021/12/11 10:21:15 No eventing-con process running
2021/12/11 10:21:15 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/11 10:21:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:21:27 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/11 10:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:21:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:21:38 Waiting for dst bucket item count to get to: 0 curr count: 856
2021/12/11 10:21:43 Waiting for dst bucket item count to get to: 0 curr count: 344
2021/12/11 10:21:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:21:43 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:21:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/11 10:21:50 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:21:55 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:22:00 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:22:05 Waiting for app: TestCountersDecrement to get deployed
2021/12/11 10:22:05 App: TestCountersDecrement got deployed
2021/12/11 10:22:05 Starting bucket ops to source bucket
2021/12/11 10:22:11 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/11 10:22:16 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/11 10:22:16 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/11 10:22:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:22:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:22:21 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:22:26 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:22:31 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:22:36 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/11 10:22:36 App: TestCountersDecrement got un-deployed
2021/12/11 10:22:36 No eventing-con process running
2021/12/11 10:22:36 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/11 10:22:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:22:48 Waiting for dst bucket item count to get to: 0 curr count: 1170
2021/12/11 10:22:53 Waiting for dst bucket item count to get to: 0 curr count: 146
2021/12/11 10:22:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:22:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:23:04 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 10:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:23:05 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/11 10:23:10 Rebalance progress: 100
--- PASS: TestCountersDecrement (177.92s)
=== RUN   TestCacheGet
2021/12/11 10:23:10 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:23:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/11 10:23:16 Waiting for app: TestCacheGet to get deployed
2021/12/11 10:23:21 Waiting for app: TestCacheGet to get deployed
2021/12/11 10:23:26 Waiting for app: TestCacheGet to get deployed
2021/12/11 10:23:31 Waiting for app: TestCacheGet to get deployed
2021/12/11 10:23:31 App: TestCacheGet got deployed
2021/12/11 10:23:31 Starting bucket ops to source bucket
2021/12/11 10:23:36 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/11 10:23:41 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/11 10:23:46 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/11 10:23:51 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/11 10:23:56 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/11 10:24:01 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/11 10:24:01 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/11 10:24:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:24:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:24:06 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 10:24:11 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 10:24:16 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 10:24:21 Waiting for app: TestCacheGet to get un-deployed
2021/12/11 10:24:21 App: TestCacheGet got un-deployed
2021/12/11 10:24:21 No eventing-con process running
2021/12/11 10:24:21 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/11 10:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:24:33 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/11 10:24:38 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/11 10:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:24:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:24:50 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/11 10:24:55 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/11 10:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.10s)
=== RUN   TestCacheRYOW
2021/12/11 10:24:55 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:24:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/11 10:25:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 10:25:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 10:25:11 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 10:25:16 Waiting for app: TestCacheRYOW to get deployed
2021/12/11 10:25:16 App: TestCacheRYOW got deployed
2021/12/11 10:25:16 Starting bucket ops to source bucket
2021/12/11 10:25:21 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/11 10:25:21 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/11 10:25:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:25:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:25:26 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 10:25:31 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 10:25:36 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 10:25:41 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/11 10:25:41 App: TestCacheRYOW got un-deployed
2021/12/11 10:25:41 No eventing-con process running
2021/12/11 10:25:41 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/11 10:25:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:25:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:25:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/11 10:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:26:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:26:10 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/11 10:26:15 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/11 10:26:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.09s)
=== RUN   TestCacheExpiry
2021/12/11 10:26:15 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:26:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/11 10:26:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 10:26:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 10:26:31 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 10:26:36 Waiting for app: TestCacheExpiry to get deployed
2021/12/11 10:26:36 App: TestCacheExpiry got deployed
2021/12/11 10:26:36 Starting bucket ops to source bucket
2021/12/11 10:26:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:26:46 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/11 10:26:51 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/11 10:26:56 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/11 10:27:01 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/11 10:27:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:27:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:27:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:27:06 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 10:27:11 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 10:27:16 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 10:27:21 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/11 10:27:21 App: TestCacheExpiry got un-deployed
2021/12/11 10:27:21 No eventing-con process running
2021/12/11 10:27:21 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/11 10:27:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:27:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:27:38 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/11 10:27:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:27:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:27:50 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/11 10:27:55 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/11 10:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.13s)
=== RUN   TestCacheExpiryCustom
2021/12/11 10:27:55 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:27:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/11 10:28:01 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 10:28:06 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 10:28:11 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 10:28:16 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/11 10:28:16 App: TestCacheExpiryCustom got deployed
2021/12/11 10:28:16 Starting bucket ops to source bucket
2021/12/11 10:28:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:28:26 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/11 10:28:31 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/11 10:28:36 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/11 10:28:41 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/11 10:28:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:28:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:28:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:28:47 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 10:28:52 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 10:28:57 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 10:29:02 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/11 10:29:02 App: TestCacheExpiryCustom got un-deployed
2021/12/11 10:29:02 No eventing-con process running
2021/12/11 10:29:02 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/11 10:29:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:29:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:29:18 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/11 10:29:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:29:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:29:30 Waiting for dst bucket item count to get to: 0 curr count: 69
2021/12/11 10:29:35 Waiting for dst bucket item count to get to: 0 curr count: 69
2021/12/11 10:29:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.30s)
=== RUN   TestCacheOverflowCustom
2021/12/11 10:29:35 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:29:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/11 10:29:41 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 10:29:46 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 10:29:51 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 10:29:56 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/11 10:29:56 App: TestCacheOverflowCustom got deployed
2021/12/11 10:29:56 Starting bucket ops to source bucket
2021/12/11 10:30:02 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/11 10:30:07 Waiting for dst bucket item count to get to: 35 curr count: 8
2021/12/11 10:30:12 Waiting for dst bucket item count to get to: 35 curr count: 8
2021/12/11 10:30:17 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/11 10:30:22 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/11 10:30:22 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/11 10:30:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:30:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:30:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 10:30:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 10:30:37 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 10:30:42 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/11 10:30:42 App: TestCacheOverflowCustom got un-deployed
2021/12/11 10:30:42 No eventing-con process running
2021/12/11 10:30:42 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/11 10:30:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:30:54 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/11 10:30:59 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/11 10:30:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:31:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:31:10 Waiting for dst bucket item count to get to: 0 curr count: 27
2021/12/11 10:31:15 Waiting for dst bucket item count to get to: 0 curr count: 27
2021/12/11 10:31:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.30s)
=== RUN   TestStrictMode
2021/12/11 10:31:15 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:31:17 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.07s)
=== RUN   TestDataTypes
2021/12/11 10:31:17 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:31:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/11 10:31:23 Waiting for app: TestDataTypes to get deployed
2021/12/11 10:31:28 Waiting for app: TestDataTypes to get deployed
2021/12/11 10:31:33 Waiting for app: TestDataTypes to get deployed
2021/12/11 10:31:38 Waiting for app: TestDataTypes to get deployed
2021/12/11 10:31:38 App: TestDataTypes got deployed
2021/12/11 10:31:38 Starting bucket ops to source bucket
2021/12/11 10:31:43 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/11 10:31:48 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/11 10:31:53 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/11 10:31:53 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/11 10:31:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:31:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:31:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 10:32:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 10:32:08 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 10:32:13 Waiting for app: TestDataTypes to get un-deployed
2021/12/11 10:32:13 App: TestDataTypes got un-deployed
2021/12/11 10:32:13 No eventing-con process running
2021/12/11 10:32:13 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/11 10:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:32:25 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 10:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:32:37 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/11 10:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.09s)
=== RUN   TestEnoentGet
2021/12/11 10:32:37 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:32:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/11 10:32:43 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:32:48 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:32:53 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:32:58 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:32:58 App: TestEnoentGet got deployed
2021/12/11 10:32:58 Starting bucket ops to source bucket
2021/12/11 10:33:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:33:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:33:13 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:33:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:33:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:33:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:33:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:33:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:33:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:33:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:33:33 App: TestEnoentGet got un-deployed
2021/12/11 10:33:33 No eventing-con process running
2021/12/11 10:33:33 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/11 10:33:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:33:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:33:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:33:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:33:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:33:57 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:33:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/11 10:34:03 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:34:08 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:34:13 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:34:18 Waiting for app: TestEnoentGet to get deployed
2021/12/11 10:34:18 App: TestEnoentGet got deployed
2021/12/11 10:34:18 Starting bucket ops to source bucket
2021/12/11 10:34:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:34:28 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:34:33 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:34:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:34:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:34:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:34:38 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:34:43 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:34:48 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:34:53 Waiting for app: TestEnoentGet to get un-deployed
2021/12/11 10:34:53 App: TestEnoentGet got un-deployed
2021/12/11 10:34:53 No eventing-con process running
2021/12/11 10:34:53 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/11 10:34:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:35:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:35:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:35:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.14s)
=== RUN   TestEnoentDelete
2021/12/11 10:35:17 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:35:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 10:35:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:35:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:35:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:35:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:35:38 App: TestEnoentDelete got deployed
2021/12/11 10:35:38 Starting bucket ops to source bucket
2021/12/11 10:35:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:35:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:35:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:35:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:35:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:35:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:35:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:36:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:36:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:36:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:36:13 App: TestEnoentDelete got un-deployed
2021/12/11 10:36:13 No eventing-con process running
2021/12/11 10:36:13 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/11 10:36:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:36:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:36:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:36:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:36:37 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:36:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 10:36:43 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:36:48 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:36:53 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:36:58 Waiting for app: TestEnoentDelete to get deployed
2021/12/11 10:36:58 App: TestEnoentDelete got deployed
2021/12/11 10:36:58 Starting bucket ops to source bucket
2021/12/11 10:37:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:37:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 10:37:08 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:37:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:37:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:37:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:37:18 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:37:23 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:37:29 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/11 10:37:29 App: TestEnoentDelete got un-deployed
2021/12/11 10:37:29 No eventing-con process running
2021/12/11 10:37:29 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/11 10:37:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:37:40 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/11 10:37:45 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/11 10:37:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:37:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:37:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.29s)
=== RUN   TestError
2021/12/11 10:38:07 Waiting for app: TestError to get un-deployed
2021/12/11 10:38:07 App: TestError got un-deployed
2021/12/11 10:38:07 No eventing-con process running
2021/12/11 10:38:07 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/11 10:38:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:38:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:38:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:38:21 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:38:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/11 10:38:27 Waiting for app: TestError to get deployed
2021/12/11 10:38:32 Waiting for app: TestError to get deployed
2021/12/11 10:38:37 Waiting for app: TestError to get deployed
2021/12/11 10:38:42 Waiting for app: TestError to get deployed
2021/12/11 10:38:42 App: TestError got deployed
2021/12/11 10:38:42 Starting bucket ops to source bucket
2021/12/11 10:38:47 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/11 10:38:52 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/11 10:38:52 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/11 10:38:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:38:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:38:57 Waiting for app: TestError to get un-deployed
2021/12/11 10:39:02 Waiting for app: TestError to get un-deployed
2021/12/11 10:39:07 Waiting for app: TestError to get un-deployed
2021/12/11 10:39:12 Waiting for app: TestError to get un-deployed
2021/12/11 10:39:12 App: TestError got un-deployed
2021/12/11 10:39:12 No eventing-con process running
2021/12/11 10:39:12 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/11 10:39:14 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:39:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:39:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:39:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:39:31 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:39:41 Waiting for dst bucket item count to get to: 0 curr count: 268
2021/12/11 10:39:46 Waiting for dst bucket item count to get to: 0 curr count: 268
2021/12/11 10:39:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.69s)
=== RUN   TestCRLF
2021/12/11 10:39:56 Waiting for app: TestCRLF to get un-deployed
2021/12/11 10:39:56 App: TestCRLF got un-deployed
2021/12/11 10:39:56 No eventing-con process running
2021/12/11 10:39:56 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/11 10:39:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:40:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:40:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:40:09 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:40:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/11 10:40:16 Waiting for app: TestCRLF to get deployed
2021/12/11 10:40:21 Waiting for app: TestCRLF to get deployed
2021/12/11 10:40:26 Waiting for app: TestCRLF to get deployed
2021/12/11 10:40:31 Waiting for app: TestCRLF to get deployed
2021/12/11 10:40:31 App: TestCRLF got deployed
2021/12/11 10:40:31 Starting bucket ops to source bucket
2021/12/11 10:40:36 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:40:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 10:40:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 10:40:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:40:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:40:46 Waiting for app: TestCRLF to get un-deployed
2021/12/11 10:40:51 Waiting for app: TestCRLF to get un-deployed
2021/12/11 10:40:56 Waiting for app: TestCRLF to get un-deployed
2021/12/11 10:41:01 Waiting for app: TestCRLF to get un-deployed
2021/12/11 10:41:01 App: TestCRLF got un-deployed
2021/12/11 10:41:01 No eventing-con process running
2021/12/11 10:41:01 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/11 10:41:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:41:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 10:41:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/11 10:41:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:41:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:41:29 Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/11 10:41:34 Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/11 10:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.75s)
=== RUN   TestImportExport
2021/12/11 10:41:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 10:41:45 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/11 10:41:45 No eventing-con process running
2021/12/11 10:41:45 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/11 10:41:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:41:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:41:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:42:03 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 10:42:03 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/11 10:42:03 No eventing-con process running
2021/12/11 10:42:03 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/11 10:42:05 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:42:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:42:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:42:17 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:42:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 10:42:18 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:42:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 10:42:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 10:42:29 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 10:42:34 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 10:42:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/11 10:42:39 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/11 10:42:44 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/11 10:42:49 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/11 10:42:54 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/11 10:42:54 App: TestImportExport_bucket_op_on_update got deployed
2021/12/11 10:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:42:59 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 10:43:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 10:43:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 10:43:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 10:43:14 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/11 10:43:14 No eventing-con process running
2021/12/11 10:43:14 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/11 10:43:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:43:19 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 10:43:24 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 10:43:29 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 10:43:35 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 10:43:35 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/11 10:43:35 No eventing-con process running
2021/12/11 10:43:35 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/11 10:43:52 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/11 10:43:52 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/11 10:43:52 No eventing-con process running
2021/12/11 10:43:52 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/11 10:43:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:43:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:44:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:44:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:44:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/11 10:44:10 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/11 10:44:10 No eventing-con process running
2021/12/11 10:44:11 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/11 10:44:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:44:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:44:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.49s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/11 10:44:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:44:34 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 10:44:34 No eventing-con process running
2021/12/11 10:44:34 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/11 10:44:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:44:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:44:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:44:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:44:48 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:44:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:44:49 Starting bucket ops to source bucket
2021/12/11 10:44:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:45:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:45:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:45:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:45:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 10:45:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:45:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 10:45:22 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 10:45:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:45:22 Undeploying app: bucket_op_on_update
2021/12/11 10:45:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:45:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:45:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:45:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:45:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:45:42 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 10:45:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:45:42 No eventing-con process running
2021/12/11 10:45:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:45:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:46:26 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:46:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:46:27 Starting bucket ops to source bucket
2021/12/11 10:46:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:46:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:46:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:46:50 Waiting for dst bucket item count to get to: 5000 curr count: 938
2021/12/11 10:46:55 Waiting for dst bucket item count to get to: 5000 curr count: 938
2021/12/11 10:46:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:47:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 10:47:00 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 10:47:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:47:00 Undeploying app: bucket_op_on_update
2021/12/11 10:47:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:47:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:47:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:47:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:47:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:47:20 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 10:47:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:47:20 No eventing-con process running
2021/12/11 10:47:22 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:47:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:48:04 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:48:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:48:05 Starting bucket ops to source bucket
2021/12/11 10:48:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:48:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:48:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:48:28 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 10:48:33 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 10:48:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:48:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 10:48:38 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 10:48:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:48:38 Undeploying app: bucket_op_on_update
2021/12/11 10:48:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:48:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:48:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:48:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:48:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:48:58 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 10:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:48:59 No eventing-con process running
2021/12/11 10:49:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:49:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:49:42 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:49:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:49:44 Starting bucket ops to source bucket
2021/12/11 10:49:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:49:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:50:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:50:07 Waiting for dst bucket item count to get to: 5000 curr count: 2388
2021/12/11 10:50:12 Waiting for dst bucket item count to get to: 5000 curr count: 4888
2021/12/11 10:50:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:50:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 10:50:17 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 10:50:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:50:17 Undeploying app: bucket_op_on_update
2021/12/11 10:50:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:50:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:50:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:50:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:50:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:50:37 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 10:50:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:50:37 No eventing-con process running
2021/12/11 10:50:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:50:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:51:21 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:51:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:51:22 Starting bucket ops to source bucket
2021/12/11 10:51:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:51:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:51:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:51:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:51:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:51:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/11 10:51:50 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/11 10:51:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:51:50 Undeploying app: bucket_op_on_update
2021/12/11 10:51:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:51:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:52:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:52:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:52:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/11 10:52:10 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/11 10:52:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:52:10 No eventing-con process running
2021/12/11 10:52:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:52:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:52:54 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.03s)
=== RUN   TestOnUpdateN1QLOp
2021/12/11 10:53:04 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 10:53:04 App: TestOnUpdateN1QLOp got un-deployed
2021/12/11 10:53:04 No eventing-con process running
2021/12/11 10:53:04 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/11 10:53:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:53:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:53:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:53:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:53:18 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:53:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/11 10:53:19 Starting bucket ops to source bucket
2021/12/11 10:53:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:53:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:53:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:53:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:53:47 Waiting for dst bucket item count to get to: 5000 curr count: 2537
2021/12/11 10:53:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:53:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:53:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:53:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 10:53:57 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 10:54:02 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 10:54:07 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/11 10:54:07 App: TestOnUpdateN1QLOp got un-deployed
2021/12/11 10:54:07 No eventing-con process running
2021/12/11 10:54:07 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/11 10:54:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:54:19 Waiting for dst bucket item count to get to: 0 curr count: 4724
2021/12/11 10:54:24 Waiting for dst bucket item count to get to: 0 curr count: 2224
2021/12/11 10:54:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:54:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:54:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 10:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.49s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/11 10:54:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 10:54:46 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/11 10:54:46 No eventing-con process running
2021/12/11 10:54:46 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/11 10:54:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:54:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:54:54 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:54:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:54:59 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:55:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:55:00 Starting bucket ops to source bucket
2021/12/11 10:55:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:55:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:55:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:55:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:55:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:55:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:55:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:55:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 10:55:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 10:55:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 10:55:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/11 10:55:43 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/11 10:55:43 No eventing-con process running
2021/12/11 10:55:43 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/11 10:55:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:55:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 10:55:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:55:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:56:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 10:56:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (91.58s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/11 10:56:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 10:56:17 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 10:56:17 No eventing-con process running
2021/12/11 10:56:17 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/11 10:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:56:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:56:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:56:31 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:56:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 10:56:32 Starting bucket ops to source bucket
2021/12/11 10:56:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:56:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:56:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:56:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:56:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:56:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:56:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:57:00 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 10:57:05 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 10:57:10 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 10:57:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 10:57:15 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 10:57:15 No eventing-con process running
2021/12/11 10:57:15 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/11 10:57:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:57:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 10:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:57:39 Waiting for dst bucket item count to get to: 0 curr count: 4531
2021/12/11 10:57:44 Waiting for dst bucket item count to get to: 0 curr count: 2031
2021/12/11 10:57:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.55s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/11 10:57:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 10:57:54 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/11 10:57:54 No eventing-con process running
2021/12/11 10:57:54 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/11 10:57:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:58:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:58:02 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:58:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:58:07 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:58:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/11 10:58:08 Starting bucket ops to source bucket
2021/12/11 10:58:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:58:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:58:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:58:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 10:58:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 10:58:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 10:58:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 10:58:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 10:58:42 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 10:58:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 10:58:52 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 10:58:57 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/11 10:58:57 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/11 10:58:57 No eventing-con process running
2021/12/11 10:58:57 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/11 10:58:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:59:08 Waiting for dst bucket item count to get to: 0 curr count: 4046
2021/12/11 10:59:13 Waiting for dst bucket item count to get to: 0 curr count: 1546
2021/12/11 10:59:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:59:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:59:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 10:59:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.54s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/11 10:59:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 10:59:35 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/11 10:59:35 No eventing-con process running
2021/12/11 10:59:35 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/11 10:59:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:59:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:59:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 10:59:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 10:59:49 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/11 10:59:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/11 10:59:50 Starting bucket ops to source bucket
2021/12/11 10:59:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:00:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:00:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:00:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:00:19 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 11:00:19 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:00:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:00:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:00:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 11:00:29 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 11:00:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 11:00:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/11 11:00:39 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/11 11:00:39 No eventing-con process running
2021/12/11 11:00:39 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/11 11:00:41 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:00:51 Waiting for dst bucket item count to get to: 0 curr count: 4317
2021/12/11 11:00:56 Waiting for dst bucket item count to get to: 0 curr count: 4317
2021/12/11 11:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:00:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:01:07 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 11:01:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.29s)
=== RUN   TestOnDeleteBucketOp
2021/12/11 11:01:17 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 11:01:17 App: TestOnDeleteBucketOp got un-deployed
2021/12/11 11:01:18 No eventing-con process running
2021/12/11 11:01:18 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/11 11:01:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:01:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:01:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:01:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:01:31 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:01:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 11:01:32 Starting bucket ops to source bucket
2021/12/11 11:01:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:01:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:01:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:01:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:01:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:01:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:01:58 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 11:02:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 11:02:08 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 11:02:13 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/11 11:02:13 App: TestOnDeleteBucketOp got un-deployed
2021/12/11 11:02:13 No eventing-con process running
2021/12/11 11:02:13 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/11 11:02:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:02:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:02:22 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:02:32 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:02:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:02:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.70s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/11 11:02:47 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 11:02:47 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/11 11:02:47 No eventing-con process running
2021/12/11 11:02:47 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/11 11:02:49 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:02:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:02:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:03:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:03:01 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:03:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/11 11:03:02 Starting bucket ops to source bucket
2021/12/11 11:03:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:03:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:03:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:03:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:03:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:03:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:03:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 11:03:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 11:03:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 11:03:43 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/11 11:03:43 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/11 11:03:43 No eventing-con process running
2021/12/11 11:03:43 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/11 11:03:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:03:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:03:52 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:04:02 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:04:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.74s)
=== RUN   TestTimerBucketOp
2021/12/11 11:04:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 11:04:17 App: TestTimerBucketOp got un-deployed
2021/12/11 11:04:17 No eventing-con process running
2021/12/11 11:04:17 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/11 11:04:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:04:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:04:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:04:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:04:30 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:04:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 11:04:32 Starting bucket ops to source bucket
2021/12/11 11:04:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:04:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:04:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:04:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:04:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:05:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:05:09 Waiting for dst bucket item count to get to: 5000 curr count: 1424
2021/12/11 11:05:14 Waiting for dst bucket item count to get to: 5000 curr count: 1424
2021/12/11 11:05:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:05:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:05:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:05:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 11:05:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 11:05:30 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 11:05:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/11 11:05:35 App: TestTimerBucketOp got un-deployed
2021/12/11 11:05:35 No eventing-con process running
2021/12/11 11:05:35 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/11 11:05:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:05:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:05:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:05:58 Waiting for dst bucket item count to get to: 0 curr count: 3873
2021/12/11 11:06:03 Waiting for dst bucket item count to get to: 0 curr count: 1373
2021/12/11 11:06:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.27s)
=== RUN   TestTimerInPastBucketOp
2021/12/11 11:06:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 11:06:13 App: TestTimerInPastBucketOp got un-deployed
2021/12/11 11:06:13 No eventing-con process running
2021/12/11 11:06:13 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/11 11:06:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:06:22 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:06:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:06:27 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:06:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 11:06:28 Starting bucket ops to source bucket
2021/12/11 11:06:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:06:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:06:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:06:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:06:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:07:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:07:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:07:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:07:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:07:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:07:11 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 11:07:16 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 11:07:21 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 11:07:26 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/11 11:07:26 App: TestTimerInPastBucketOp got un-deployed
2021/12/11 11:07:26 No eventing-con process running
2021/12/11 11:07:26 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/11 11:07:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:07:38 Waiting for dst bucket item count to get to: 0 curr count: 2944
2021/12/11 11:07:43 Waiting for dst bucket item count to get to: 0 curr count: 444
2021/12/11 11:07:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:07:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:07:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.41s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/11 11:08:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:08:05 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 11:08:05 No eventing-con process running
2021/12/11 11:08:05 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/11 11:08:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:08:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:08:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:08:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:08:18 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:08:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 11:08:19 Starting bucket ops to source bucket
2021/12/11 11:08:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:08:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:08:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:08:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:08:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:08:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:08:57 Waiting for dst bucket item count to get to: 5000 curr count: 1486
2021/12/11 11:09:02 Waiting for dst bucket item count to get to: 5000 curr count: 3986
2021/12/11 11:09:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:09:07 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 11:09:07 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 11:09:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:09:07 Undeploying app: bucket_op_with_timer
2021/12/11 11:09:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:09:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:09:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:09:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:09:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:09:27 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 11:09:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:09:27 No eventing-con process running
2021/12/11 11:09:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:09:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:10:11 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:10:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 11:10:12 Starting bucket ops to source bucket
2021/12/11 11:10:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:10:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:10:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:10:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:10:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:10:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:10:50 Waiting for dst bucket item count to get to: 5000 curr count: 616
2021/12/11 11:10:55 Waiting for dst bucket item count to get to: 5000 curr count: 616
2021/12/11 11:10:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:11:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 11:11:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 11:11:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:11:00 Undeploying app: bucket_op_with_timer
2021/12/11 11:11:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:11:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:11:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:11:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:11:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:11:21 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 11:11:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:11:21 No eventing-con process running
2021/12/11 11:11:22 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:11:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:12:05 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:12:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 11:12:06 Starting bucket ops to source bucket
2021/12/11 11:12:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:12:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:12:49 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 11:12:49 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 11:12:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:12:49 Undeploying app: bucket_op_with_timer
2021/12/11 11:12:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:12:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:12:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:13:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:13:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:13:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 11:13:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:13:09 No eventing-con process running
2021/12/11 11:13:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:13:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:13:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:13:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 11:13:54 Starting bucket ops to source bucket
2021/12/11 11:14:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:14:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:14:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:14:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:14:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:14:27 Waiting for dst bucket item count to get to: 5000 curr count: 59
2021/12/11 11:14:32 Waiting for dst bucket item count to get to: 5000 curr count: 169
2021/12/11 11:14:37 Waiting for dst bucket item count to get to: 5000 curr count: 2610
2021/12/11 11:14:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:14:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 11:14:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 11:14:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:14:42 Undeploying app: bucket_op_with_timer
2021/12/11 11:14:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:14:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:14:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:14:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:15:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:15:02 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 11:15:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:15:02 No eventing-con process running
2021/12/11 11:15:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:15:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:15:46 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:15:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 11:15:47 Starting bucket ops to source bucket
2021/12/11 11:15:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:16:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:16:30 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/11 11:16:30 App: TestDeployUndeployLoopTimer got deployed
2021/12/11 11:16:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:16:30 Undeploying app: bucket_op_with_timer
2021/12/11 11:16:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:16:36 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:16:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:16:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:16:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/11 11:16:51 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/11 11:16:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:16:51 No eventing-con process running
2021/12/11 11:16:52 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:17:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:17:35 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.00s)
=== RUN   TestMultipleHandlers
2021/12/11 11:17:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 11:17:45 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/11 11:17:45 No eventing-con process running
2021/12/11 11:17:45 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/11 11:17:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:17:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:17:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:17:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:18:03 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 11:18:03 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/11 11:18:03 No eventing-con process running
2021/12/11 11:18:03 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/11 11:18:05 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:18:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:18:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:18:17 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:18:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 11:18:18 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:18:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/11 11:18:24 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 11:18:29 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 11:18:34 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 11:18:39 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/11 11:18:39 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/11 11:18:44 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/11 11:18:49 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/11 11:18:54 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/11 11:18:54 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/11 11:18:54 Starting bucket ops to source bucket
2021/12/11 11:19:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:19:09 Waiting for dst bucket item count to get to: 5000 curr count: 4085
2021/12/11 11:19:09 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:19:09 Starting bucket ops to source bucket
2021/12/11 11:19:24 Waiting for dst bucket item count to get to: 10000 curr count: 6707
2021/12/11 11:19:29 Waiting for dst bucket item count to get to: 10000 curr count: 9203
2021/12/11 11:19:29 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 11:19:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:19:29 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:19:29 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:19:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:19:34 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 11:19:39 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 11:19:44 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 11:19:49 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/11 11:19:49 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/11 11:19:49 No eventing-con process running
2021/12/11 11:19:49 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/11 11:19:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:19:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:19:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:20:08 Waiting for dst bucket item count to get to: 0 curr count: 7424
2021/12/11 11:20:13 Waiting for dst bucket item count to get to: 0 curr count: 2428
2021/12/11 11:20:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:20:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:20:19 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 11:20:24 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 11:20:29 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 11:20:34 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/11 11:20:34 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/11 11:20:34 No eventing-con process running
2021/12/11 11:20:34 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/11 11:20:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:20:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:20:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:20:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.61s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/11 11:20:57 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 11:20:57 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/11 11:20:57 No eventing-con process running
2021/12/11 11:20:57 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/11 11:20:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:21:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:21:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:21:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:21:11 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:21:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 11:21:12 Starting bucket ops to source bucket
2021/12/11 11:21:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:21:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:21:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:21:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:21:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:21:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:21:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 11:21:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 11:21:40 Pausing app: bucket_op_on_update
2021/12/11 11:21:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:21:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 11:21:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 11:21:45 Resuming app: bucket_op_on_update
2021/12/11 11:21:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:21:46 Starting bucket ops to source bucket
2021/12/11 11:21:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:21:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:22:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:22:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 11:22:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:22:09 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 11:22:09 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 11:22:09 Pausing app: bucket_op_on_update
2021/12/11 11:22:09 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:22:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 11:22:14 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 11:22:14 Resuming app: bucket_op_on_update
2021/12/11 11:22:15 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:22:15 Starting bucket ops to source bucket
2021/12/11 11:22:23 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 11:22:28 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 11:22:33 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 11:22:33 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/11 11:22:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:22:38 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 11:22:38 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 11:22:38 Pausing app: bucket_op_on_update
2021/12/11 11:22:39 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:22:44 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 11:22:44 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 11:22:44 Resuming app: bucket_op_on_update
2021/12/11 11:22:45 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:22:45 Starting bucket ops to source bucket
2021/12/11 11:22:53 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:22:58 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:23:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:23:08 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/11 11:23:08 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/11 11:23:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:23:13 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 11:23:13 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 11:23:13 Pausing app: bucket_op_on_update
2021/12/11 11:23:13 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:23:18 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 11:23:18 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 11:23:18 Resuming app: bucket_op_on_update
2021/12/11 11:23:19 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:23:19 Starting bucket ops to source bucket
2021/12/11 11:23:27 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:23:32 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:23:37 Waiting for dst bucket item count to get to: 25000 curr count: 22168
2021/12/11 11:23:42 Waiting for dst bucket item count to get to: 25000 curr count: 24668
2021/12/11 11:23:42 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/11 11:23:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:23:47 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/11 11:23:47 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/11 11:23:47 Pausing app: bucket_op_on_update
2021/12/11 11:23:47 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:23:52 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/11 11:23:52 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/11 11:23:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:23:57 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 11:24:02 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 11:24:07 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 11:24:12 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/11 11:24:12 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/11 11:24:12 No eventing-con process running
2021/12/11 11:24:12 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/11 11:24:14 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:24:24 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 11:24:29 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/11 11:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:24:31 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:24:41 Waiting for dst bucket item count to get to: 0 curr count: 22926
2021/12/11 11:24:46 Waiting for dst bucket item count to get to: 0 curr count: 22926
2021/12/11 11:24:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (238.58s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/11 11:24:56 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 11:24:56 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/11 11:24:56 No eventing-con process running
2021/12/11 11:24:56 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/11 11:24:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:25:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:25:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:25:09 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:25:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 11:25:10 Starting bucket ops to source bucket
2021/12/11 11:25:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:25:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:25:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:25:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:25:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:25:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:25:49 Waiting for dst bucket item count to get to: 5000 curr count: 2811
2021/12/11 11:25:54 Waiting for dst bucket item count to get to: 5000 curr count: 2811
2021/12/11 11:25:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:25:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:25:59 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 11:25:59 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 11:25:59 Pausing app: bucket_op_with_timer
2021/12/11 11:25:59 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:26:04 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 11:26:04 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 11:26:04 Resuming app: bucket_op_with_timer
2021/12/11 11:26:05 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:26:05 Starting bucket ops to source bucket
2021/12/11 11:26:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:26:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:26:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:26:28 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:26:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:26:38 Waiting for dst bucket item count to get to: 10000 curr count: 8305
2021/12/11 11:26:43 Waiting for dst bucket item count to get to: 10000 curr count: 8305
2021/12/11 11:26:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 11:26:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:26:48 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 11:26:48 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 11:26:48 Pausing app: bucket_op_with_timer
2021/12/11 11:26:48 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:26:53 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 11:26:53 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 11:26:53 Resuming app: bucket_op_with_timer
2021/12/11 11:26:54 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:26:54 Starting bucket ops to source bucket
2021/12/11 11:27:02 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 11:27:07 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 11:27:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/11 11:27:17 Waiting for dst bucket item count to get to: 15000 curr count: 10986
2021/12/11 11:27:22 Waiting for dst bucket item count to get to: 15000 curr count: 10986
2021/12/11 11:27:27 Waiting for dst bucket item count to get to: 15000 curr count: 14380
2021/12/11 11:27:32 Waiting for dst bucket item count to get to: 15000 curr count: 14380
2021/12/11 11:27:32 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/11 11:27:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:27:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 11:27:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 11:27:37 Pausing app: bucket_op_with_timer
2021/12/11 11:27:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:27:43 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 11:27:43 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 11:27:43 Resuming app: bucket_op_with_timer
2021/12/11 11:27:44 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:27:44 Starting bucket ops to source bucket
2021/12/11 11:27:52 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:27:57 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:28:02 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:28:07 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/11 11:28:12 Waiting for dst bucket item count to get to: 20000 curr count: 15988
2021/12/11 11:28:17 Waiting for dst bucket item count to get to: 20000 curr count: 15988
2021/12/11 11:28:17 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/11 11:28:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:28:22 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 11:28:22 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 11:28:22 Pausing app: bucket_op_with_timer
2021/12/11 11:28:22 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:28:27 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 11:28:27 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 11:28:27 Resuming app: bucket_op_with_timer
2021/12/11 11:28:28 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:28:28 Starting bucket ops to source bucket
2021/12/11 11:28:36 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:28:41 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:28:46 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:28:51 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:28:56 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/11 11:29:01 Waiting for dst bucket item count to get to: 25000 curr count: 20161
2021/12/11 11:29:06 Waiting for dst bucket item count to get to: 25000 curr count: 22661
2021/12/11 11:29:06 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/11 11:29:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:29:11 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/11 11:29:11 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/11 11:29:11 Pausing app: bucket_op_with_timer
2021/12/11 11:29:11 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:29:16 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/11 11:29:16 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/11 11:29:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:29:21 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 11:29:26 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 11:29:31 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 11:29:36 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/11 11:29:36 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/11 11:29:36 No eventing-con process running
2021/12/11 11:29:36 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/11 11:29:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:29:48 Waiting for dst bucket item count to get to: 0 curr count: 18028
2021/12/11 11:29:53 Waiting for dst bucket item count to get to: 0 curr count: 5528
2021/12/11 11:29:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:29:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:30:05 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 11:30:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.15s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/11 11:30:15 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 11:30:15 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/11 11:30:15 No eventing-con process running
2021/12/11 11:30:15 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/11 11:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:30:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:30:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:30:28 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:30:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/11 11:30:35 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 11:30:40 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 11:30:45 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 11:30:50 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/11 11:30:50 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/11 11:30:50 Starting bucket ops to source bucket
2021/12/11 11:30:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:31:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:31:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:31:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:31:19 Waiting for dst bucket item count to get to: 5000 curr count: 2282
2021/12/11 11:31:24 Waiting for dst bucket item count to get to: 5000 curr count: 4561
2021/12/11 11:31:29 Waiting for dst bucket item count to get to: 5000 curr count: 4779
2021/12/11 11:31:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:31:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:31:29 Pausing app: bucket_op_with_timer
2021/12/11 11:31:29 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:31:34 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/11 11:31:34 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/11 11:31:34 Resuming app: bucket_op_with_timer
2021/12/11 11:31:35 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:31:35 Starting bucket ops to source bucket
2021/12/11 11:31:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:31:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:31:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:32:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:32:07 Waiting for dst bucket item count to get to: 10000 curr count: 7389
2021/12/11 11:32:12 Waiting for dst bucket item count to get to: 10000 curr count: 7389
2021/12/11 11:32:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 11:32:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:32:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 11:32:22 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 11:32:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 11:32:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/11 11:32:32 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/11 11:32:32 No eventing-con process running
2021/12/11 11:32:32 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/11 11:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:32:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 11:32:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:32:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:32:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:33:00 Waiting for dst bucket item count to get to: 0 curr count: 8147
2021/12/11 11:33:05 Waiting for dst bucket item count to get to: 0 curr count: 8147
2021/12/11 11:33:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (180.52s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/11 11:33:15 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 11:33:15 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/11 11:33:15 No eventing-con process running
2021/12/11 11:33:15 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/11 11:33:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:33:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:33:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:33:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:33:29 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:33:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/11 11:33:35 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 11:33:40 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 11:33:45 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 11:33:50 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/11 11:33:50 App: TestPauseResumeWithEventingReb got deployed
2021/12/11 11:33:50 Starting bucket ops to source bucket
2021/12/11 11:33:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:34:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:34:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:34:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:34:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:34:15 Pausing app: bucket_op_with_timer
2021/12/11 11:34:15 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:34:20 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/11 11:34:20 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/11 11:34:20 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 11:34:20 Removing node(s): [] from the cluster
2021/12/11 11:34:25 Rebalance progress: 29.16666666666666
2021/12/11 11:34:30 Rebalance progress: 29.16666666666666
2021/12/11 11:34:35 Rebalance progress: 100
2021/12/11 11:34:35 Resuming app: bucket_op_with_timer
2021/12/11 11:34:36 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:34:36 Starting bucket ops to source bucket
2021/12/11 11:34:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:34:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:34:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:35:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/11 11:35:08 Waiting for dst bucket item count to get to: 10000 curr count: 6414
2021/12/11 11:35:13 Waiting for dst bucket item count to get to: 10000 curr count: 6414
2021/12/11 11:35:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 11:35:13 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 11:35:18 Rebalance progress: 29.16666666666666
2021/12/11 11:35:23 Rebalance progress: 29.16666666666666
2021/12/11 11:35:28 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/11 11:35:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:35:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:36:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:36:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:36:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:36:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:37:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:37:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:37:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 11:37:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:80} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9e5cb8fd4702a6936d50595c5f05f7df
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 54bfc1322a753e6ae2b48b43a8551c6d
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 9b0afd3b39399cf5a06c7a9b1127a44b
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/11 11:38:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:38:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:38:08 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 11:38:13 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 11:38:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 11:38:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/11 11:38:23 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/11 11:38:23 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:38:23 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/11 11:38:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:38:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 11:38:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:38:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:38:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 11:38:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (341.44s)
=== RUN   TestBucketDeleteAfterPause
2021/12/11 11:38:57 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/11 11:38:57 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/11 11:38:57 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:38:57 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/11 11:38:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:39:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:39:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:39:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:39:11 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:39:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/11 11:39:17 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 11:39:22 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 11:39:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 11:39:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/11 11:39:32 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/11 11:39:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:39:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:39:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:39:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:39:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:39:46 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/11 11:39:46 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:39:51 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/11 11:39:51 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/11 11:39:51 handler_test.go:716: Deleting source bucket: default
2021/12/11 11:39:58 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/11 11:39:58 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/11 11:40:04 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/11 11:40:04 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/11 11:40:04 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:40:04 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/11 11:40:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:40:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 11:40:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:40:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:40:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:40:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.85s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/11 11:40:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 11:40:37 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/11 11:40:37 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:40:37 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/11 11:40:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:40:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:40:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:40:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:40:50 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:40:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/11 11:40:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 11:41:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 11:41:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 11:41:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/11 11:41:11 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/11 11:41:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:41:46 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/11 11:41:46 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:41:51 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/11 11:41:51 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/11 11:44:51 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/11 11:44:51 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/11 11:44:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/11 11:44:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 11:44:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:44:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/11 11:45:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 11:45:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:45:02 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2474 expected 5000
2021/12/11 11:45:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 11:45:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:45:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3050 expected 5000
2021/12/11 11:45:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/11 11:45:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:45:12 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/11 11:45:12 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/11 11:45:12 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:45:22 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 11:45:27 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 11:45:32 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/11 11:45:32 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/11 11:45:33 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:45:33 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/11 11:45:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:45:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:45:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 11:45:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:45:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:45:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.77s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/11 11:46:06 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 11:46:06 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/11 11:46:06 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:46:06 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/11 11:46:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:46:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:46:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:46:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:46:19 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:46:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 11:46:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:46:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:46:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:46:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:46:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/11 11:46:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:46:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:46:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:46:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/11 11:46:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:46:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:46:59 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/11 11:46:59 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:47:29 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/11 11:47:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 11:47:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:47:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:47:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:47:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/11 11:47:50 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/11 11:47:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:48:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:48:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 369
2021/12/11 11:48:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 369
2021/12/11 11:48:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:48:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:48:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/11 11:48:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:48:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 11:48:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 11:48:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/11 11:48:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/11 11:48:33 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:48:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/11 11:48:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:48:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:48:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:48:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (170.39s)
=== RUN   TestCPPWorkerCleanup
2021/12/11 11:48:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 11:48:56 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/11 11:48:56 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:48:56 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/11 11:48:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:49:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:49:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:49:09 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:49:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/11 11:49:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:49:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:49:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:49:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:49:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:49:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4878
2021/12/11 11:49:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4878
2021/12/11 11:49:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:49:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:49:42 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:49:47 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 11:49:52 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 11:49:57 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 11:50:02 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/11 11:50:02 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/11 11:50:02 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:50:02 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/11 11:50:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:50:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:50:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 11:50:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:50:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:50:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4056
2021/12/11 11:50:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4056
2021/12/11 11:50:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.51s)
=== RUN   TestWithUserXattrs
2021/12/11 11:51:15 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 11:51:15 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/11 11:51:15 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:51:15 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/11 11:51:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:51:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:51:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:51:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:51:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:51:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:51:32 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:51:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/11 11:51:38 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 11:51:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 11:51:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 11:51:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/11 11:51:54 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/11 11:51:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:52:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:52:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2150
2021/12/11 11:52:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4650
2021/12/11 11:52:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:52:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:52:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 11:52:18 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 11:52:23 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 11:52:28 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 11:52:33 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/11 11:52:33 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/11 11:52:33 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:52:33 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/11 11:52:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:52:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 11:52:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 11:52:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:52:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:53:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4979
2021/12/11 11:53:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4979
2021/12/11 11:53:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (120.58s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/11 11:53:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 11:53:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/11 11:53:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:53:16 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/11 11:53:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:53:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:53:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:53:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:53:29 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:53:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/11 11:53:35 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 11:53:40 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 11:53:45 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 11:53:50 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/11 11:53:50 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/11 11:53:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:53:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:54:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 11:54:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 11:54:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:54:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:54:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 11:54:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:54:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:54:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3553
2021/12/11 11:54:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3553
2021/12/11 11:54:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:54:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 11:55:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15744
2021/12/11 11:55:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15744
2021/12/11 11:55:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/11 11:55:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 11:55:06 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 11:55:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 11:55:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 11:55:21 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 11:55:26 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 11:55:26 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/11 11:55:26 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:55:31 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/11 11:55:31 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/11 11:55:31 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:55:31 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/11 11:55:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:55:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 11:55:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/11 11:55:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:55:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:55:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24642
2021/12/11 11:56:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12142
2021/12/11 11:56:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.86s)
=== RUN   TestMetaBucketDelete
2021/12/11 11:56:14 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/11 11:56:14 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/11 11:56:14 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:56:14 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/11 11:56:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:56:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:56:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:56:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:56:27 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:56:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 11:56:33 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 11:56:38 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 11:56:43 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 11:56:48 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/11 11:56:48 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/11 11:56:48 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/11 11:56:50 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/11 11:56:55 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/11 11:56:55 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/11 11:57:10 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/11 11:57:10 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/11 11:57:10 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:57:10 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/11 11:57:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:57:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:57:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:57:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.93s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/11 11:57:33 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 11:57:33 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 11:57:33 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:57:33 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/11 11:57:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:57:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:57:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:57:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:57:46 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:57:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 11:58:06 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/11 11:58:06 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/11 11:58:21 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 11:58:21 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 11:58:36 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 11:58:36 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 11:58:36 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:58:36 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/11 11:58:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:58:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:58:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:58:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.62s)
=== RUN   TestSourceBucketDelete
2021/12/11 11:58:59 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/11 11:58:59 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/11 11:58:59 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:58:59 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/11 11:59:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:59:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:59:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 11:59:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 11:59:12 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/11 11:59:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/11 11:59:19 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 11:59:24 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 11:59:29 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 11:59:34 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/11 11:59:34 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/11 11:59:34 handler_test.go:945: Deleting source bucket: default
2021/12/11 11:59:35 handler_test.go:947: Deleted source bucket: default
2021/12/11 11:59:40 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/11 11:59:40 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/11 11:59:55 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/11 11:59:55 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/11 11:59:55 eventingSetup.go:46: No eventing-con process running
2021/12/11 11:59:55 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/11 11:59:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:00:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:00:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:00:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.16s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/11 12:00:19 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/11 12:00:19 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/11 12:00:19 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:00:19 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/11 12:00:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:00:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:00:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:00:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:00:32 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:00:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 12:00:52 handler_test.go:966: Deleting source bucket: default
2021/12/11 12:00:53 handler_test.go:968: Deleted source bucket: default
2021/12/11 12:01:08 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/11 12:01:08 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/11 12:01:23 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/11 12:01:23 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/11 12:01:23 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:01:23 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/11 12:01:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:01:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:01:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:01:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.62s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/11 12:01:46 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 12:01:46 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 12:01:46 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:01:46 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/11 12:01:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:01:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:01:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:01:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:01:59 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:02:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 12:02:19 handler_test.go:988: Deleting source bucket: default
2021/12/11 12:02:20 handler_test.go:990: Deleted source bucket: default
2021/12/11 12:02:20 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/11 12:02:21 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/11 12:02:36 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 12:02:36 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 12:02:51 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/11 12:02:51 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/11 12:02:51 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:02:51 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/11 12:02:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:02:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:02:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:03:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (87.79s)
=== RUN   TestUndeployDuringBootstrap
2021/12/11 12:03:14 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 12:03:14 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/11 12:03:14 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:03:14 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/11 12:03:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:03:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:03:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:03:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:03:27 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:03:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 12:03:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 12:03:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:03:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:03:40 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/11 12:03:40 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/11 12:03:40 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/11 12:03:45 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/11 12:03:50 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/11 12:03:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 12:04:00 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 12:04:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 12:04:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/11 12:04:05 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:04:30 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/11 12:04:30 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/11 12:04:30 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:04:30 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/11 12:04:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:04:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:04:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:04:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:04:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:04:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3599
2021/12/11 12:05:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1099
2021/12/11 12:05:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.29s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/11 12:05:08 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:05:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/11 12:05:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 12:05:19 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 12:05:24 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 12:05:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/11 12:05:29 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/11 12:05:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/11 12:05:30 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 12:05:30 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/11 12:05:35 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/11 12:05:35 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/11 12:05:35 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:05:35 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/11 12:05:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:05:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:05:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:05:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.57s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/11 12:05:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 12:05:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/11 12:05:58 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:05:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/11 12:05:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:06:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:06:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:06:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:06:11 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:06:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/11 12:06:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 12:06:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 12:06:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 12:06:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/11 12:06:32 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/11 12:06:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 12:07:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:07:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 12:07:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 12:07:18 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 12:07:23 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 12:07:23 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/11 12:07:23 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:09:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/11 12:09:08 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/11 12:09:08 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:09:08 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/11 12:09:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:09:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:09:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 12:09:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:09:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:09:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:09:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (227.99s)
=== RUN   TestUndeployWithKVFailover
2021/12/11 12:09:46 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/11 12:09:46 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/11 12:09:46 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:09:46 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/11 12:09:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:09:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:09:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:09:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:09:59 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:10:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/11 12:10:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 12:10:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:10:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:10:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:10:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:10:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:10:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2
2021/12/11 12:10:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3617
2021/12/11 12:10:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3617
2021/12/11 12:10:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 12:10:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 12:10:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 12:10:48 Rebalance progress: 29.16666666666666
2021/12/11 12:10:53 Rebalance progress: 29.16666666666666
2021/12/11 12:10:58 Rebalance progress: 100
2021/12/11 12:10:58 Failing over: [127.0.0.1:19003]
2021/12/11 12:11:08 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/11 12:12:08 Removing node(s): [] from the cluster
2021/12/11 12:12:13 Rebalance progress: 29.16666666666666
2021/12/11 12:12:18 Rebalance progress: 29.16666666666666
2021/12/11 12:12:23 Rebalance progress: 100
2021/12/11 12:12:28 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/11 12:12:28 App: TestUndeployWithKVFailover got un-deployed
2021/12/11 12:12:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:12:33 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/11 12:12:33 App: TestUndeployWithKVFailover got un-deployed
2021/12/11 12:12:33 No eventing-con process running
2021/12/11 12:12:33 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/11 12:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:12:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:12:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 12:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:12:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:13:01 Waiting for dst bucket item count to get to: 0 curr count: 4889
2021/12/11 12:13:06 Waiting for dst bucket item count to get to: 0 curr count: 4889
2021/12/11 12:13:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.16s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/11 12:13:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 12:13:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/11 12:13:16 No eventing-con process running
2021/12/11 12:13:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/11 12:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:13:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:13:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:13:29 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:13:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/11 12:13:35 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 12:13:40 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 12:13:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 12:13:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/11 12:13:50 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/11 12:13:50 Starting bucket ops to source bucket
2021/12/11 12:14:10 Waiting for dst bucket item count to get to: 20000 curr count: 10451
2021/12/11 12:14:10 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/11 12:14:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:14:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:14:23 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/11 12:14:28 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 12:14:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:14:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:14:40 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 12:14:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 12:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:14:45 Starting bucket ops to source bucket
2021/12/11 12:14:57 Waiting for dst bucket item count to get to: 10000 curr count: 4537
2021/12/11 12:15:02 Waiting for dst bucket item count to get to: 10000 curr count: 4537
2021/12/11 12:15:02 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 12:15:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:15:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:15:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 12:15:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 12:15:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 12:15:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/11 12:15:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/11 12:15:23 No eventing-con process running
2021/12/11 12:15:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/11 12:15:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:15:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 12:15:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:15:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:15:41 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:15:51 Waiting for dst bucket item count to get to: 0 curr count: 8671
2021/12/11 12:15:56 Waiting for dst bucket item count to get to: 0 curr count: 8671
2021/12/11 12:15:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (169.64s)
=== RUN   TestTimerOverwrite
2021/12/11 12:16:06 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 12:16:06 App: TestTimerOverwrite got un-deployed
2021/12/11 12:16:06 No eventing-con process running
2021/12/11 12:16:06 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/11 12:16:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:16:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:16:14 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:16:19 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:16:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/11 12:16:25 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 12:16:30 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 12:16:35 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 12:16:40 Waiting for app: TestTimerOverwrite to get deployed
2021/12/11 12:16:40 App: TestTimerOverwrite got deployed
2021/12/11 12:16:50 Starting bucket ops to source bucket
2021/12/11 12:16:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:17:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 12:18:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:18:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:18:09 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 12:18:14 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 12:18:19 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 12:18:24 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/11 12:18:24 App: TestTimerOverwrite got un-deployed
2021/12/11 12:18:24 No eventing-con process running
2021/12/11 12:18:24 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/11 12:18:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:18:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:18:37 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:18:47 Waiting for dst bucket item count to get to: 0 curr count: 2512
2021/12/11 12:18:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (171.74s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/11 12:18:52 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 12:18:53 Removing node(s): [] from the cluster
2021/12/11 12:18:58 Rebalance progress: 100
2021/12/11 12:18:58 Failing over: [127.0.0.1:19003]
2021/12/11 12:19:03 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/11 12:19:03 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/11 12:19:03 No eventing-con process running
2021/12/11 12:19:03 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/11 12:19:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:19:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:19:11 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:19:16 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:19:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/11 12:19:17 Starting bucket ops to source bucket
2021/12/11 12:19:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:19:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:19:27 Context: Post to set retry counter, response code: 200 dump: 
2021/12/11 12:19:27 set retry counter, response &{[] }
2021/12/11 12:20:32 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/11 12:20:32 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/11 12:20:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:20:32 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/11 12:20:37 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/11 12:20:37 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/11 12:20:37 No eventing-con process running
2021/12/11 12:20:37 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/11 12:20:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:20:48 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:20:53 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 12:20:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:20:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:21:05 Waiting for dst bucket item count to get to: 0 curr count: 3606
2021/12/11 12:21:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:21:05 Removing node(s): [] from the cluster
2021/12/11 12:21:10 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.78s)
=== RUN   TestOnUpdateSrcMutation
2021/12/11 12:21:20 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 12:21:20 App: TestOnUpdateSrcMutation got un-deployed
2021/12/11 12:21:20 No eventing-con process running
2021/12/11 12:21:20 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/11 12:21:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:21:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:21:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:21:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:21:33 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:21:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/11 12:21:34 Starting bucket ops to source bucket
2021/12/11 12:22:01 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 12:22:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:22:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:22:07 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 12:22:12 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 12:22:17 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 12:22:22 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/11 12:22:22 App: TestOnUpdateSrcMutation got un-deployed
2021/12/11 12:22:22 No eventing-con process running
2021/12/11 12:22:22 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/11 12:22:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:22:33 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 12:22:38 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:22:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:22:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.47s)
=== RUN   TestOnDeleteSrcMutation
2021/12/11 12:22:55 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 12:22:55 App: TestOnDeleteSrcMutation got un-deployed
2021/12/11 12:22:55 No eventing-con process running
2021/12/11 12:22:55 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/11 12:22:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:23:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:23:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:23:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:23:08 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:23:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/11 12:23:09 Starting bucket ops to source bucket
2021/12/11 12:23:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 12:23:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:23:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:23:43 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 12:23:48 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 12:23:53 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 12:23:58 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/11 12:23:58 App: TestOnDeleteSrcMutation got un-deployed
2021/12/11 12:23:58 No eventing-con process running
2021/12/11 12:23:58 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/11 12:23:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:24:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:24:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 12:24:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:24:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:24:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.20s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/11 12:24:31 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 12:24:31 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/11 12:24:31 No eventing-con process running
2021/12/11 12:24:31 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/11 12:24:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:24:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:24:39 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:24:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:24:44 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:24:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 12:24:45 Starting bucket ops to source bucket
2021/12/11 12:25:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 12:25:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:25:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:25:27 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 12:25:32 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 12:25:37 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 12:25:42 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/11 12:25:42 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/11 12:25:42 No eventing-con process running
2021/12/11 12:25:42 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/11 12:25:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:25:53 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 12:25:58 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:26:00 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.41s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/11 12:26:15 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 12:26:15 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/11 12:26:15 No eventing-con process running
2021/12/11 12:26:15 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/11 12:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:26:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:26:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:26:28 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:26:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/11 12:26:30 Starting bucket ops to source bucket
2021/12/11 12:27:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 12:27:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:27:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:27:18 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 12:27:23 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 12:27:28 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 12:27:33 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/11 12:27:33 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/11 12:27:33 No eventing-con process running
2021/12/11 12:27:33 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/11 12:27:35 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:27:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 12:27:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:27:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:27:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.04s)
=== RUN   TestInterHandlerRecursion
2021/12/11 12:28:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:28:01 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 12:28:01 No eventing-con process running
2021/12/11 12:28:01 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/11 12:28:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:28:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:28:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:28:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:28:19 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:28:19 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 12:28:19 No eventing-con process running
2021/12/11 12:28:19 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/11 12:28:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:28:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:28:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:28:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:28:32 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:28:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/11 12:28:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:28:39 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:28:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:28:49 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:28:54 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:28:54 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/11 12:28:54 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:28:54 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/11 12:28:54 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/11 12:28:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:28:59 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:29:04 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:29:09 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:29:14 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:29:14 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 12:29:14 No eventing-con process running
2021/12/11 12:29:14 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/11 12:29:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:29:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:29:22 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:29:32 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:29:32 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 12:29:32 No eventing-con process running
2021/12/11 12:29:32 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/11 12:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:29:40 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.67s)
=== RUN   TestInterBucketRecursion
2021/12/11 12:29:50 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:29:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 12:29:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:29:51 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:29:51 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/11 12:29:56 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 12:30:01 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 12:30:06 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 12:30:11 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/11 12:30:11 App: TestInterBucketRecursion_function1 got deployed
2021/12/11 12:30:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:30:16 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:30:21 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:30:26 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:30:31 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:30:31 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/11 12:30:31 No eventing-con process running
2021/12/11 12:30:31 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/11 12:30:33 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:30:39 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:30:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:30:49 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/11 12:30:49 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/11 12:30:49 No eventing-con process running
2021/12/11 12:30:49 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/11 12:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:30:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:31:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.45s)
=== RUN   TestLargeHandler
2021/12/11 12:31:02 Changed force_compress value to false
2021/12/11 12:31:02 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:31:03 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/11 12:31:03 Changed force_compress value to true
2021/12/11 12:31:03 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:31:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/11 12:31:10 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 12:31:15 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 12:31:20 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 12:31:25 Waiting for app: TestLargeHandler_function to get deployed
2021/12/11 12:31:25 App: TestLargeHandler_function got deployed
2021/12/11 12:31:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:31:30 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 12:31:35 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 12:31:40 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 12:31:45 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/11 12:31:45 App: TestLargeHandler_function got un-deployed
2021/12/11 12:31:45 No eventing-con process running
2021/12/11 12:31:45 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/11 12:31:46 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:31:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.43s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/11 12:32:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:32:08 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 12:32:08 No eventing-con process running
2021/12/11 12:32:08 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/11 12:32:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:32:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:32:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:32:26 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:32:26 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 12:32:26 No eventing-con process running
2021/12/11 12:32:26 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/11 12:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:32:39 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:32:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/11 12:32:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:32:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:32:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:32:55 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:33:00 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/11 12:33:00 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/11 12:33:00 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:33:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/11 12:33:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:33:07 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 12:33:12 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 12:33:17 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 12:33:22 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/11 12:33:22 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/11 12:33:22 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/11 12:33:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:33:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:33:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:33:37 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:33:42 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/11 12:33:42 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/11 12:33:42 No eventing-con process running
2021/12/11 12:33:42 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/11 12:33:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:33:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:33:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:34:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:34:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:34:11 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:34:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/11 12:34:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/11 12:34:16 No eventing-con process running
2021/12/11 12:34:16 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/11 12:34:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:34:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:34:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.15s)
=== RUN   TestAllowInterBucketRecursion
2021/12/11 12:34:34 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:34:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 12:34:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:34:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 12:34:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 12:34:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 12:34:55 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/11 12:34:55 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/11 12:34:55 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:34:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/11 12:34:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:35:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 12:35:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 12:35:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 12:35:16 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/11 12:35:16 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/11 12:35:16 TestAllowInterBucketRecursion_function2 is deployed
2021/12/11 12:35:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:35:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:35:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:35:32 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:35:37 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/11 12:35:37 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/11 12:35:37 No eventing-con process running
2021/12/11 12:35:37 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/11 12:35:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:35:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:35:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:35:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:35:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:35:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 12:36:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 12:36:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 12:36:11 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/11 12:36:11 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/11 12:36:11 No eventing-con process running
2021/12/11 12:36:11 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/11 12:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:36:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.91s)
=== RUN   TestInterBucketRecursion2
2021/12/11 12:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:36:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:36:42 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:36:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:36:53 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:36:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 12:36:59 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 12:37:04 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 12:37:09 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 12:37:14 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/11 12:37:14 App: TestInterBucketRecursion2_function1 got deployed
2021/12/11 12:37:14 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:37:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/11 12:37:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 12:37:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 12:37:31 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 12:37:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/11 12:37:36 App: TestInterBucketRecursion2_function2 got deployed
2021/12/11 12:37:36 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:37:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/11 12:37:42 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 12:37:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 12:37:52 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 12:37:57 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/11 12:37:57 App: TestInterBucketRecursion2_function3 got deployed
2021/12/11 12:37:57 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:37:57 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/11 12:37:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:38:02 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 12:38:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 12:38:12 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 12:38:17 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/11 12:38:17 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/11 12:38:17 No eventing-con process running
2021/12/11 12:38:17 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/11 12:38:18 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:38:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:38:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:38:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:38:35 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 12:38:40 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 12:38:45 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 12:38:50 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/11 12:38:50 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/11 12:38:50 No eventing-con process running
2021/12/11 12:38:50 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/11 12:38:52 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:38:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:39:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:39:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:39:09 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 12:39:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 12:39:19 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 12:39:24 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/11 12:39:24 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/11 12:39:24 No eventing-con process running
2021/12/11 12:39:24 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/11 12:39:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:39:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:39:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:39:42 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/11 12:39:42 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/11 12:39:42 No eventing-con process running
2021/12/11 12:39:42 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/11 12:39:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:39:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:39:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:39:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.16s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/11 12:40:02 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 12:40:07 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/11 12:40:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:40:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:40:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:40:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:41:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:41:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:41:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:41:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:42:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:42:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:42:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:42:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:43:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:43:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:43:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:43:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:44:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:44:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:44:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:44:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:45:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:45:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:45:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/11 12:45:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/11 12:46:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:46:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 12:46:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 12:46:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/11 12:46:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:46:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 12:46:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 12:46:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/11 12:46:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:46:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/11 12:46:26 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/11 12:46:26 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:46:26 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/11 12:46:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:46:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:46:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:46:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:46:44 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:46:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 12:46:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 12:46:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 12:47:00 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 12:47:05 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/11 12:47:05 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9e5cb8fd4702a6936d50595c5f05f7df
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 54bfc1322a753e6ae2b48b43a8551c6d
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
Producer node: 127.0.0.1:9303	Node UUID: 9b0afd3b39399cf5a06c7a9b1127a44b
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/11 12:47:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:47:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 12:47:10 Rebalance progress: 29.16666666666666
2021/12/11 12:47:15 Rebalance progress: 29.16666666666666
2021/12/11 12:47:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9e5cb8fd4702a6936d50595c5f05f7df
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: 54bfc1322a753e6ae2b48b43a8551c6d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 9b0afd3b39399cf5a06c7a9b1127a44b
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]


2021/12/11 12:47:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:47:20 handler_test.go:1674: Deleting source bucket: default
2021/12/11 12:47:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 12:47:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 12:47:27 Rebalance progress: 29.16666666666666
2021/12/11 12:47:32 Rebalance progress: 29.16666666666666
2021/12/11 12:47:37 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 898
	vb list dump: [0-3, 5-11, 13-14, 17-44, 46, 48-53, 55-57, 59-72, 74-81, 83, 85-86, 88-112, 114-139, 141-144, 146-156, 158-160, 162-165, 167-175, 178-179, 181-185, 187-198, 200, 202-205, 207, 209-221, 223-224, 226-231, 234-236, 238, 240-269, 272-285, 287-313, 315-316, 318, 320-321, 323-325, 327-337, 339-341, 343, 345-349, 351-352, 354-367, 370-373, 375-411, 413-429, 431-435, 437, 439-442, 444-449, 451-454, 456, 458-460, 462-467, 469, 472, 474, 477-481, 483-485, 487-489, 492, 494-495, 498-501, 503-504, 506-590, 592, 594-603, 606-616, 618-627, 629-636, 638-640, 642-650, 652, 655-660, 662-680, 682-697, 699-713, 716-721, 723-734, 736-740, 742-747, 749-765, 767-783, 785, 787-789, 793-813, 815-816, 818-827, 829-841, 843-847, 850-870, 872, 874-885, 887, 889-890, 892-897, 899, 901-912, 914-916, 918-926, 928-931, 933-941, 943-949, 951-962, 964-965, 967-972, 975-980, 982-983, 986-987, 989-1000, 1002-1023]
	stream status: stopped
	len: 126
	vb list dump: [4, 12, 15-16, 45, 47, 54, 58, 73, 82, 84, 87, 113, 140, 145, 157, 161, 166, 176-177, 180, 186, 199, 201, 206, 208, 222, 225, 232-233, 237, 239, 270-271, 286, 314, 317, 319, 322, 326, 338, 342, 344, 350, 353, 368-369, 374, 412, 430, 436, 438, 443, 450, 455, 457, 461, 468, 470-471, 473, 475-476, 482, 486, 490-491, 493, 496-497, 502, 505, 591, 593, 604-605, 617, 628, 637, 641, 651, 653-654, 661, 681, 698, 714-715, 722, 735, 741, 748, 766, 784, 786, 790-792, 814, 817, 828, 842, 848-849, 871, 873, 886, 888, 891, 898, 900, 913, 917, 927, 932, 942, 950, 963, 966, 973-974, 981, 984-985, 988, 1001]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9e5cb8fd4702a6936d50595c5f05f7df
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-3, 5-11, 13-14, 17-44, 46, 48-53, 55-57, 59-72, 74-81, 83, 85-86, 88-112]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [114-139, 141-144, 146-156, 158-160, 162-165, 167-175, 178-179, 181-185, 187-198, 200, 202-205, 207, 209-221, 223-224, 226-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-231, 234-236, 238, 240-269, 272-285, 287-313, 315-316, 318, 320-321, 323-325, 327-337, 339-341]
Producer node: 172.31.2.134:9300	Node UUID: 54bfc1322a753e6ae2b48b43a8551c6d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 98
	v: [911-912, 914-916, 918-926, 928-931, 933-941, 943-949, 951-962, 964-965, 967-972, 975-980, 982-983, 986-987, 989-1000, 1002-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-697, 699-713, 716-721, 723-734, 736-740, 742-747, 749-765, 767-783, 785, 787-789, 793-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-813, 815-816, 818-827, 829-841, 843-847, 850-870, 872, 874-885, 887, 889-890, 892-897, 899, 901-910]
Producer node: 127.0.0.1:9303	Node UUID: 9b0afd3b39399cf5a06c7a9b1127a44b
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [343, 345-349, 351-352, 354-367, 370-373, 375-411, 413-429, 431-435, 437, 439-442, 444-449, 451-454]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 97
	v: [456, 458-460, 462-467, 469, 472, 474, 477-481, 483-485, 487-489, 492, 494-495, 498-501, 503-504, 506-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-590, 592, 594-603, 606-616, 618-627, 629-636, 638-640, 642-650, 652, 655-660, 662-680, 682]


2021/12/11 12:47:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:47:43 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/11 12:47:43 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/11 12:47:48 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/11 12:47:48 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/11 12:47:48 eventingSetup.go:46: No eventing-con process running
2021/12/11 12:47:48 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/11 12:47:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:47:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:47:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:48:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:48:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/11 12:48:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 12:48:06 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (491.77s)
=== RUN   TestN1QLRecursion
2021/12/11 12:48:09 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 12:48:09 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/11 12:48:09 Removing node(s): [] from the cluster
2021/12/11 12:48:14 Rebalance progress: 100
2021/12/11 12:48:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:48:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:48:37 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:48:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 12:48:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:48:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:48:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:48:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:48:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:49:04 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:49:09 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:49:14 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/11 12:49:14 App: TestN1QLRecursion_function1 got deployed
2021/12/11 12:49:14 TestN1QLRecursion_function1 is deployed
2021/12/11 12:49:14 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:49:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/11 12:49:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:49:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 12:49:25 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 12:49:30 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 12:49:35 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/11 12:49:35 App: TestN1QLRecursion_function3 got deployed
2021/12/11 12:49:35 TestN1QLRecursion_function3 is deployed
2021/12/11 12:49:35 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:49:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/11 12:49:35 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/11 12:49:35 Success: TestN1QLRecursion_function2 is not deployed
2021/12/11 12:49:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:49:40 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 12:49:45 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 12:49:50 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 12:49:55 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/11 12:49:55 App: TestN1QLRecursion_function1 got un-deployed
2021/12/11 12:49:55 No eventing-con process running
2021/12/11 12:49:55 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/11 12:49:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:50:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:50:04 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:50:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:50:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:50:14 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 12:50:19 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 12:50:24 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 12:50:29 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/11 12:50:29 App: TestN1QLRecursion_function3 got un-deployed
2021/12/11 12:50:29 No eventing-con process running
2021/12/11 12:50:29 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/11 12:50:31 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:50:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:50:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:50:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:50:44 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 12:50:49 Rebalance progress: 100
--- PASS: TestN1QLRecursion (160.30s)
=== RUN   TestN1QLAllowRecursion
2021/12/11 12:50:49 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 12:50:50 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/11 12:50:50 Removing node(s): [] from the cluster
2021/12/11 12:50:55 Rebalance progress: 100
2021/12/11 12:51:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:51:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:51:17 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:51:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/11 12:51:18 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:51:23 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 12:51:28 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 12:51:33 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 12:51:38 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/11 12:51:38 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/11 12:51:38 TestN1QLAllowRecursion_function1 is deployed
2021/12/11 12:51:38 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:51:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/11 12:51:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:51:45 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 12:51:50 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 12:51:55 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 12:52:00 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/11 12:52:00 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/11 12:52:00 TestN1QLAllowRecursion_function3 is deployed
2021/12/11 12:52:00 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:52:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/11 12:52:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/11 12:52:06 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 12:52:11 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 12:52:16 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 12:52:21 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/11 12:52:21 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/11 12:52:21 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/11 12:52:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:52:26 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 12:52:31 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 12:52:36 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 12:52:41 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/11 12:52:41 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/11 12:52:41 No eventing-con process running
2021/12/11 12:52:41 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/11 12:52:43 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:52:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:52:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:52:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:53:00 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 12:53:05 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 12:53:10 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 12:53:15 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/11 12:53:15 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/11 12:53:15 No eventing-con process running
2021/12/11 12:53:15 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/11 12:53:17 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:53:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:53:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:53:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:53:34 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 12:53:39 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 12:53:44 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 12:53:49 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/11 12:53:49 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/11 12:53:49 No eventing-con process running
2021/12/11 12:53:49 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/11 12:53:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:53:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:53:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:54:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:54:04 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 12:54:09 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.85s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/11 12:54:09 Sleeping for some time
2021/12/11 12:54:19 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 12:54:19 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/11 12:54:19 No eventing-con process running
2021/12/11 12:54:19 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/11 12:54:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:54:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:54:28 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:54:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:54:33 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:54:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 12:54:34 Starting bucket ops to source bucket
2021/12/11 12:55:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 12:55:28 Waiting for dst bucket item count to get to: 5000 curr count: 544
2021/12/11 12:55:33 Waiting for dst bucket item count to get to: 5000 curr count: 3044
2021/12/11 12:55:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 12:55:33 Starting bucket ops to source bucket
2021/12/11 12:55:34 Starting bucket ops to source bucket
2021/12/11 12:56:40 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/11 12:56:40 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/11 12:56:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:56:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:56:45 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 12:56:50 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 12:56:55 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 12:57:00 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/11 12:57:00 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/11 12:57:00 No eventing-con process running
2021/12/11 12:57:00 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/11 12:57:02 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:57:12 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 12:57:17 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 12:57:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:57:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:57:29 Waiting for dst bucket item count to get to: 0 curr count: 6361
2021/12/11 12:57:34 Waiting for dst bucket item count to get to: 0 curr count: 2861
2021/12/11 12:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.75s)
=== RUN   TestTimerOverWriteSameReference
2021/12/11 12:57:39 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 12:57:39 App: TestTimerOverWriteSameReference got un-deployed
2021/12/11 12:57:39 No eventing-con process running
2021/12/11 12:57:39 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/11 12:57:41 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:57:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:57:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 12:57:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 12:57:52 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/11 12:57:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/11 12:57:59 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 12:58:04 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 12:58:09 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 12:58:14 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/11 12:58:14 App: TestTimerOverWriteSameReference got deployed
2021/12/11 12:58:24 Starting bucket ops to source bucket
2021/12/11 12:59:25 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/11 12:59:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 12:59:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 12:59:40 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 12:59:45 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 12:59:50 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 12:59:55 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/11 12:59:55 App: TestTimerOverWriteSameReference got un-deployed
2021/12/11 12:59:55 No eventing-con process running
2021/12/11 12:59:55 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/11 12:59:57 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:00:07 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 13:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:00:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:00:18 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/11 13:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (164.67s)
=== RUN   TestCancelTimerBucketop
2021/12/11 13:00:23 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 13:00:23 App: TestCancelTimerBucketop got un-deployed
2021/12/11 13:00:23 No eventing-con process running
2021/12/11 13:00:23 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/11 13:00:25 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:00:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:00:32 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:00:37 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:00:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/11 13:00:43 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 13:00:48 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 13:00:53 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 13:00:58 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/11 13:00:58 App: TestCancelTimerBucketop got deployed
2021/12/11 13:00:58 Starting bucket ops to source bucket
2021/12/11 13:01:05 Starting bucket ops to source bucket
2021/12/11 13:02:08 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 13:02:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:02:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:02:13 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 13:02:18 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 13:02:23 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 13:02:28 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/11 13:02:28 App: TestCancelTimerBucketop got un-deployed
2021/12/11 13:02:28 No eventing-con process running
2021/12/11 13:02:28 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/11 13:02:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:02:40 Waiting for dst bucket item count to get to: 0 curr count: 518
2021/12/11 13:02:45 Waiting for dst bucket item count to get to: 0 curr count: 518
2021/12/11 13:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:02:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:02:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 13:02:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.29s)
=== RUN   TestJSExpiryDate
2021/12/11 13:03:02 Starting bucket ops to source bucket
2021/12/11 13:03:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 13:03:15 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/11 13:03:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/11 13:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:03:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:03:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:03:51 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 13:03:56 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 13:04:01 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 13:04:06 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/11 13:04:06 App: TestJSExpiryDate got un-deployed
2021/12/11 13:04:06 No eventing-con process running
2021/12/11 13:04:06 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/11 13:04:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:04:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:04:15 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:04:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.70s)
=== RUN   TestBinaryDoc
2021/12/11 13:04:25 Starting bucket ops to source bucket
2021/12/11 13:04:25 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:04:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/11 13:04:27 Starting bucket ops to source bucket
2021/12/11 13:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:04:27 Deleting source bucket docs now
2021/12/11 13:04:27 Starting bucket ops to source bucket
2021/12/11 13:05:04 Waiting for dst bucket item count to get to: 32 curr count: 21
2021/12/11 13:05:09 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/11 13:05:14 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/11 13:05:14 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/11 13:05:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:05:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:05:19 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 13:05:24 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 13:05:29 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 13:05:34 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/11 13:05:34 App: TestBinaryDoc got un-deployed
2021/12/11 13:05:34 No eventing-con process running
2021/12/11 13:05:34 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/11 13:05:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:05:46 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/11 13:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:05:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:05:58 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/11 13:05:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.30s)
=== RUN   TestConstantBindings
2021/12/11 13:05:58 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:05:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/11 13:05:59 Starting bucket ops to source bucket
2021/12/11 13:06:04 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 13:06:09 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 13:06:14 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 13:06:19 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/11 13:06:24 Waiting for dst bucket item count to get to: 6000 curr count: 3000
2021/12/11 13:06:24 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/11 13:06:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:06:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:06:30 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 13:06:35 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 13:06:40 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 13:06:45 Waiting for app: TestConstantBindings to get un-deployed
2021/12/11 13:06:45 App: TestConstantBindings got un-deployed
2021/12/11 13:06:45 No eventing-con process running
2021/12/11 13:06:45 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/11 13:06:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:06:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 13:06:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:07:08 Waiting for dst bucket item count to get to: 0 curr count: 4906
2021/12/11 13:07:13 Waiting for dst bucket item count to get to: 0 curr count: 1906
2021/12/11 13:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (75.63s)
=== RUN   TestCodeInsight
2021/12/11 13:07:18 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 13:07:18 App: TestCodeInsight got un-deployed
2021/12/11 13:07:18 No eventing-con process running
2021/12/11 13:07:18 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/11 13:07:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:07:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:07:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:07:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:07:32 Starting bucket ops to source bucket
2021/12/11 13:07:33 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:07:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/11 13:07:39 Waiting for app: TestCodeInsight to get deployed
2021/12/11 13:07:44 Waiting for app: TestCodeInsight to get deployed
2021/12/11 13:07:49 Waiting for app: TestCodeInsight to get deployed
2021/12/11 13:07:54 Waiting for app: TestCodeInsight to get deployed
2021/12/11 13:07:54 App: TestCodeInsight got deployed
2021/12/11 13:07:59 Waiting for dst bucket item count to get to: 1000 curr count: 951
2021/12/11 13:07:59 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 13:07:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:08:04 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 13:08:09 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 13:08:14 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 13:08:19 Waiting for app: TestCodeInsight to get un-deployed
2021/12/11 13:08:19 App: TestCodeInsight got un-deployed
2021/12/11 13:08:19 No eventing-con process running
2021/12/11 13:08:19 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/11 13:08:21 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:08:31 Waiting for dst bucket item count to get to: 0 curr count: 943
2021/12/11 13:08:36 Waiting for dst bucket item count to get to: 0 curr count: 943
2021/12/11 13:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:08:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:08:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 13:08:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (94.32s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13023.853s

Functional tests took 3 hours, 37 minutes and 6 seconds.

N1ql tests

2021/12/11 13:08:57 Started n1ql
2021/12/11 13:09:07 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/11 13:09:07 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/11 13:09:07 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 13:09:08 Removing node(s): [] from the cluster
2021/12/11 13:09:13 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/11 13:09:18 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/11 13:09:18 App: TestRecursiveMutationN1QL got un-deployed
2021/12/11 13:09:18 No eventing-con process running
2021/12/11 13:09:18 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/11 13:09:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:09:26 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:09:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:09:31 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:09:31 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.54s)
=== RUN   TestFlexReset1
2021/12/11 13:09:36 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 13:09:36 App: TestFlexReset1 got un-deployed
2021/12/11 13:09:36 No eventing-con process running
2021/12/11 13:09:36 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/11 13:09:38 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:09:45 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:09:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:09:50 Starting bucket ops to source bucket
2021/12/11 13:09:50 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:09:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/11 13:09:57 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 13:10:02 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 13:10:07 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 13:10:12 Waiting for app: TestFlexReset1 to get deployed
2021/12/11 13:10:12 App: TestFlexReset1 got deployed
2021/12/11 13:10:12 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 13:10:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:10:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:10:17 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 13:10:22 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 13:10:27 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 13:10:32 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/11 13:10:32 App: TestFlexReset1 got un-deployed
2021/12/11 13:10:32 No eventing-con process running
2021/12/11 13:10:32 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/11 13:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:10:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 13:10:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 13:10:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:10:50 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:11:00 Waiting for dst bucket item count to get to: 0 curr count: 813
2021/12/11 13:11:05 Waiting for dst bucket item count to get to: 0 curr count: 813
2021/12/11 13:11:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.14s)
=== RUN   TestFlexReset2
2021/12/11 13:11:10 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 13:11:10 App: TestFlexReset2 got un-deployed
2021/12/11 13:11:10 No eventing-con process running
2021/12/11 13:11:10 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/11 13:11:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:11:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:11:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:11:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:11:24 Starting bucket ops to source bucket
2021/12/11 13:11:25 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:11:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/11 13:11:31 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 13:11:36 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 13:11:41 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 13:11:46 Waiting for app: TestFlexReset2 to get deployed
2021/12/11 13:11:46 App: TestFlexReset2 got deployed
2021/12/11 13:11:51 Waiting for dst bucket item count to get to: 1000 curr count: 222
2021/12/11 13:11:51 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/11 13:11:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:11:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:11:56 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 13:12:01 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 13:12:06 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 13:12:11 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/11 13:12:11 App: TestFlexReset2 got un-deployed
2021/12/11 13:12:11 No eventing-con process running
2021/12/11 13:12:11 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/11 13:12:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:12:23 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/11 13:12:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 13:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:12:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:12:40 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 13:12:45 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 13:12:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.16s)
=== RUN   TestN1QLLabelledBreak
2021/12/11 13:12:50 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 13:12:50 App: TestN1QLLabelledBreak got un-deployed
2021/12/11 13:12:50 No eventing-con process running
2021/12/11 13:12:50 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/11 13:12:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:12:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:12:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:13:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:13:03 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:13:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/11 13:13:09 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 13:13:14 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 13:13:19 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 13:13:24 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/11 13:13:24 App: TestN1QLLabelledBreak got deployed
2021/12/11 13:13:24 Starting bucket ops to source bucket
2021/12/11 13:13:34 Waiting for dst bucket item count to get to: 10000 curr count: 188
2021/12/11 13:13:39 Waiting for dst bucket item count to get to: 10000 curr count: 188
2021/12/11 13:13:44 Waiting for dst bucket item count to get to: 10000 curr count: 582
2021/12/11 13:13:49 Waiting for dst bucket item count to get to: 10000 curr count: 2361
2021/12/11 13:13:54 Waiting for dst bucket item count to get to: 10000 curr count: 4374
2021/12/11 13:13:59 Waiting for dst bucket item count to get to: 10000 curr count: 6385
2021/12/11 13:14:04 Waiting for dst bucket item count to get to: 10000 curr count: 8375
2021/12/11 13:14:09 Waiting for dst bucket item count to get to: 10000 curr count: 9397
2021/12/11 13:14:09 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 13:14:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:14:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:14:14 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 13:14:19 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 13:14:24 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 13:14:29 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/11 13:14:29 App: TestN1QLLabelledBreak got un-deployed
2021/12/11 13:14:29 No eventing-con process running
2021/12/11 13:14:29 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/11 13:14:31 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:14:41 Waiting for dst bucket item count to get to: 0 curr count: 4842
2021/12/11 13:14:46 Waiting for dst bucket item count to get to: 0 curr count: 4842
2021/12/11 13:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:14:58 Waiting for dst bucket item count to get to: 0 curr count: 5575
2021/12/11 13:15:03 Waiting for dst bucket item count to get to: 0 curr count: 575
2021/12/11 13:15:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (138.10s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/11 13:15:08 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 13:15:08 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/11 13:15:08 No eventing-con process running
2021/12/11 13:15:08 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/11 13:15:10 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:15:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:15:16 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:15:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:15:21 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:15:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/11 13:15:27 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 13:15:32 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 13:15:37 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 13:15:42 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/11 13:15:42 App: TestN1QLUnlabelledBreak got deployed
2021/12/11 13:15:42 Starting bucket ops to source bucket
2021/12/11 13:15:52 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 13:15:57 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 13:16:02 Waiting for dst bucket item count to get to: 10000 curr count: 1696
2021/12/11 13:16:07 Waiting for dst bucket item count to get to: 10000 curr count: 1696
2021/12/11 13:16:12 Waiting for dst bucket item count to get to: 10000 curr count: 7367
2021/12/11 13:16:17 Waiting for dst bucket item count to get to: 10000 curr count: 7367
2021/12/11 13:16:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 13:16:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:16:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:16:22 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 13:16:27 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 13:16:32 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 13:16:37 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/11 13:16:37 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/11 13:16:37 No eventing-con process running
2021/12/11 13:16:37 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/11 13:16:39 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:16:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:16:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 13:16:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:16:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:17:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/11 13:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (123.09s)
=== RUN   TestN1QLThrowStatement
2021/12/11 13:17:11 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 13:17:11 App: TestN1QLThrowStatement got un-deployed
2021/12/11 13:17:11 No eventing-con process running
2021/12/11 13:17:11 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/11 13:17:13 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:17:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:17:19 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:17:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:17:24 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:17:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/11 13:17:30 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 13:17:36 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 13:17:41 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 13:17:46 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/11 13:17:46 App: TestN1QLThrowStatement got deployed
2021/12/11 13:17:46 Starting bucket ops to source bucket
2021/12/11 13:17:55 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/11 13:18:00 Waiting for dst bucket item count to get to: 10000 curr count: 357
2021/12/11 13:18:05 Waiting for dst bucket item count to get to: 10000 curr count: 357
2021/12/11 13:18:10 Waiting for dst bucket item count to get to: 10000 curr count: 3956
2021/12/11 13:18:15 Waiting for dst bucket item count to get to: 10000 curr count: 3956
2021/12/11 13:18:20 Waiting for dst bucket item count to get to: 10000 curr count: 8022
2021/12/11 13:18:25 Waiting for dst bucket item count to get to: 10000 curr count: 8022
2021/12/11 13:18:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/11 13:18:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:18:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:18:30 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 13:18:35 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 13:18:40 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 13:18:45 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/11 13:18:45 App: TestN1QLThrowStatement got un-deployed
2021/12/11 13:18:45 No eventing-con process running
2021/12/11 13:18:45 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/11 13:18:47 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:18:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:18:59 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:19:09 Waiting for dst bucket item count to get to: 0 curr count: 9191
2021/12/11 13:19:14 Waiting for dst bucket item count to get to: 0 curr count: 4191
2021/12/11 13:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (127.83s)
=== RUN   TestN1QLNestedForLoop
2021/12/11 13:19:19 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 13:19:19 App: TestN1QLNestedForLoop got un-deployed
2021/12/11 13:19:19 No eventing-con process running
2021/12/11 13:19:19 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/11 13:19:20 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:19:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:19:27 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:19:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:19:32 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:19:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/11 13:19:38 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 13:19:43 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 13:19:48 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 13:19:53 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/11 13:19:53 App: TestN1QLNestedForLoop got deployed
2021/12/11 13:19:53 Starting bucket ops to source bucket
2021/12/11 13:20:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 13:20:08 Waiting for dst bucket item count to get to: 5000 curr count: 134
2021/12/11 13:20:13 Waiting for dst bucket item count to get to: 5000 curr count: 445
2021/12/11 13:20:18 Waiting for dst bucket item count to get to: 5000 curr count: 1551
2021/12/11 13:20:23 Waiting for dst bucket item count to get to: 5000 curr count: 2735
2021/12/11 13:20:28 Waiting for dst bucket item count to get to: 5000 curr count: 3935
2021/12/11 13:20:33 Waiting for dst bucket item count to get to: 5000 curr count: 4940
2021/12/11 13:20:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 13:20:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:20:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:20:38 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 13:20:43 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 13:20:48 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 13:20:53 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/11 13:20:53 App: TestN1QLNestedForLoop got un-deployed
2021/12/11 13:20:53 No eventing-con process running
2021/12/11 13:20:53 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/11 13:20:55 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:21:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:21:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:21:07 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:21:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:21:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.89s)
=== RUN   TestN1QLAutoClose
2021/12/11 13:21:22 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 13:21:22 App: TestN1QLAutoClose got un-deployed
2021/12/11 13:21:22 No eventing-con process running
2021/12/11 13:21:22 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/11 13:21:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:21:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:21:35 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:21:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/11 13:21:41 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 13:21:46 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 13:21:51 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 13:21:56 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/11 13:21:56 App: TestN1QLAutoClose got deployed
2021/12/11 13:21:56 Starting bucket ops to source bucket
2021/12/11 13:22:02 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/11 13:22:07 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/11 13:22:12 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/11 13:22:17 Waiting for dst bucket item count to get to: 500 curr count: 25
2021/12/11 13:22:22 Waiting for dst bucket item count to get to: 500 curr count: 25
2021/12/11 13:22:27 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/11 13:22:32 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/11 13:22:37 Waiting for dst bucket item count to get to: 500 curr count: 69
2021/12/11 13:22:42 Waiting for dst bucket item count to get to: 500 curr count: 69
2021/12/11 13:22:47 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/11 13:22:52 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/11 13:22:57 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/11 13:23:02 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/11 13:23:07 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/11 13:23:12 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/11 13:23:17 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/11 13:23:22 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/11 13:23:27 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/11 13:23:32 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/11 13:23:37 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/11 13:23:42 Waiting for dst bucket item count to get to: 500 curr count: 200
2021/12/11 13:23:47 Waiting for dst bucket item count to get to: 500 curr count: 210
2021/12/11 13:23:52 Waiting for dst bucket item count to get to: 500 curr count: 222
2021/12/11 13:23:57 Waiting for dst bucket item count to get to: 500 curr count: 231
2021/12/11 13:24:02 Waiting for dst bucket item count to get to: 500 curr count: 241
2021/12/11 13:24:07 Waiting for dst bucket item count to get to: 500 curr count: 251
2021/12/11 13:24:12 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/11 13:24:17 Waiting for dst bucket item count to get to: 500 curr count: 272
2021/12/11 13:24:22 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/11 13:24:27 Waiting for dst bucket item count to get to: 500 curr count: 297
2021/12/11 13:24:32 Waiting for dst bucket item count to get to: 500 curr count: 304
2021/12/11 13:24:37 Waiting for dst bucket item count to get to: 500 curr count: 315
2021/12/11 13:24:42 Waiting for dst bucket item count to get to: 500 curr count: 327
2021/12/11 13:24:47 Waiting for dst bucket item count to get to: 500 curr count: 338
2021/12/11 13:24:52 Waiting for dst bucket item count to get to: 500 curr count: 347
2021/12/11 13:24:57 Waiting for dst bucket item count to get to: 500 curr count: 356
2021/12/11 13:25:02 Waiting for dst bucket item count to get to: 500 curr count: 367
2021/12/11 13:25:07 Waiting for dst bucket item count to get to: 500 curr count: 387
2021/12/11 13:25:12 Waiting for dst bucket item count to get to: 500 curr count: 387
2021/12/11 13:25:17 Waiting for dst bucket item count to get to: 500 curr count: 408
2021/12/11 13:25:22 Waiting for dst bucket item count to get to: 500 curr count: 408
2021/12/11 13:25:27 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/11 13:25:32 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/11 13:25:37 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/11 13:25:42 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/11 13:25:47 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/11 13:25:52 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/11 13:25:57 Waiting for dst bucket item count to get to: 500 curr count: 495
2021/12/11 13:26:02 Waiting for dst bucket item count to get to: 500 curr count: 495
2021/12/11 13:26:02 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/11 13:26:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:26:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:26:07 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 13:26:12 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 13:26:17 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 13:26:22 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/11 13:26:22 App: TestN1QLAutoClose got un-deployed
2021/12/11 13:26:22 No eventing-con process running
2021/12/11 13:26:22 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/11 13:26:24 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:26:34 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/11 13:26:39 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/11 13:26:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:26:41 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:26:51 Waiting for dst bucket item count to get to: 0 curr count: 480
2021/12/11 13:26:56 Waiting for dst bucket item count to get to: 0 curr count: 480
2021/12/11 13:26:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.41s)
=== RUN   TestN1QLPosParams
2021/12/11 13:27:01 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 13:27:01 App: TestN1QLPosParams got un-deployed
2021/12/11 13:27:01 No eventing-con process running
2021/12/11 13:27:01 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/11 13:27:03 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:27:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:27:09 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:27:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:27:15 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:27:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/11 13:27:21 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 13:27:26 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 13:27:31 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 13:27:36 Waiting for app: TestN1QLPosParams to get deployed
2021/12/11 13:27:36 App: TestN1QLPosParams got deployed
2021/12/11 13:27:36 Starting bucket ops to source bucket
2021/12/11 13:27:45 Waiting for dst bucket item count to get to: 5000 curr count: 810
2021/12/11 13:27:50 Waiting for dst bucket item count to get to: 5000 curr count: 810
2021/12/11 13:27:50 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 13:27:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:27:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:27:55 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 13:28:00 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 13:28:05 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 13:28:10 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/11 13:28:10 App: TestN1QLPosParams got un-deployed
2021/12/11 13:28:10 No eventing-con process running
2021/12/11 13:28:10 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/11 13:28:12 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:28:22 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:28:27 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 13:28:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:28:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:28:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:28:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 13:28:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.11s)
=== RUN   TestN1QLExhaustConnPool
2021/12/11 13:28:49 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 13:28:49 App: TestN1QLExhaustConnPool got un-deployed
2021/12/11 13:28:49 No eventing-con process running
2021/12/11 13:28:49 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/11 13:28:51 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:28:58 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:29:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:29:03 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:29:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/11 13:29:09 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 13:29:14 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 13:29:19 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 13:29:24 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/11 13:29:24 App: TestN1QLExhaustConnPool got deployed
2021/12/11 13:29:24 Starting bucket ops to source bucket
2021/12/11 13:29:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/11 13:29:34 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/11 13:29:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/11 13:29:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:29:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:29:39 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 13:29:44 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 13:29:49 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 13:29:54 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/11 13:29:54 App: TestN1QLExhaustConnPool got un-deployed
2021/12/11 13:29:54 No eventing-con process running
2021/12/11 13:29:54 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/11 13:29:56 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:30:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/11 13:30:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:30:08 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:30:18 Waiting for dst bucket item count to get to: 0 curr count: 59
2021/12/11 13:30:23 Waiting for dst bucket item count to get to: 0 curr count: 9
2021/12/11 13:30:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.78s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/11 13:30:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 13:30:28 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/11 13:30:28 No eventing-con process running
2021/12/11 13:30:28 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/11 13:30:30 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:30:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:30:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:30:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:30:41 Starting bucket ops to source bucket
2021/12/11 13:30:58 Starting bucket ops to source bucket
2021/12/11 13:31:16 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 17141
2021/12/11 13:31:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/11 13:31:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 13:31:27 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 13:31:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 13:31:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/11 13:31:37 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/11 13:31:37 Starting bucket ops to source bucket
2021/12/11 13:31:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:31:47 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:31:52 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:31:57 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:32:02 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:32:07 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:32:12 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:32:17 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:32:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/11 13:32:22 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/11 13:32:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:32:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:32:27 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 13:32:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 13:32:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 13:32:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/11 13:32:42 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/11 13:32:42 No eventing-con process running
2021/12/11 13:32:42 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/11 13:32:44 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:32:54 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/11 13:32:59 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/11 13:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:33:01 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:33:11 Waiting for dst bucket item count to get to: 0 curr count: 27692
2021/12/11 13:33:16 Waiting for dst bucket item count to get to: 0 curr count: 27692
2021/12/11 13:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (172.97s)
=== RUN   TestN1QLQueryConsistency
2021/12/11 13:33:21 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 13:33:21 App: n1qlquery_consistency got un-deployed
2021/12/11 13:33:21 No eventing-con process running
2021/12/11 13:33:21 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/11 13:33:23 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:33:29 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:33:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:33:34 Starting bucket ops to source bucket
2021/12/11 13:33:38 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:33:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/11 13:33:44 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 13:33:49 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 13:33:54 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 13:33:59 Waiting for app: n1qlquery_consistency to get deployed
2021/12/11 13:33:59 App: n1qlquery_consistency got deployed
2021/12/11 13:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/11 13:34:09 Waiting for dst bucket item count to get to: 5000 curr count: 4437
2021/12/11 13:34:14 Waiting for dst bucket item count to get to: 5000 curr count: 4437
2021/12/11 13:34:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/11 13:34:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/11 13:34:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:34:19 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 13:34:24 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 13:34:29 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 13:34:34 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/11 13:34:34 App: n1qlquery_consistency got un-deployed
2021/12/11 13:34:34 No eventing-con process running
2021/12/11 13:34:34 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/11 13:34:36 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:34:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/11 13:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:34:48 Context: Bucket flush, response code: 200 dump: 
2021/12/11 13:34:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/11 13:34:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.84s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1556.225s

N1ql tests took 0 hours, 26 minutes and 1 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/11 13:36:30 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/11 13:36:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/11 13:36:53 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 13:36:53 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/11 13:36:53 No eventing-con process running
2021/12/11 13:36:53 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/11 13:36:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:36:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:37:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:37:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:37:11 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:37:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 13:37:17 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 13:37:22 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 13:37:27 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 13:37:32 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/11 13:37:32 App: TestDuplicateWithEventingRebIn got deployed
2021/12/11 13:37:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 13:37:32 Starting bucket ops to source bucket
2021/12/11 13:37:46 Removing node(s): [] from the cluster
2021/12/11 13:37:51 Rebalance progress: 0
2021/12/11 13:37:56 Rebalance progress: 0
2021/12/11 13:38:01 Rebalance progress: 16.66666666666666
2021/12/11 13:38:06 Rebalance progress: 25
2021/12/11 13:38:11 Rebalance progress: 25
2021/12/11 13:38:16 Rebalance progress: 37.73584905660377
2021/12/11 13:38:21 Rebalance progress: 98.58490566037736
2021/12/11 13:38:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:38:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 13:38:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:38:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 13:38:31 Rebalance progress: 25
2021/12/11 13:38:36 Rebalance progress: 25
2021/12/11 13:38:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:38:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:38:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:38:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 13:38:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:38:41 Waiting for src bucket item count to get to: 200000 curr count: 93039
2021/12/11 13:38:46 Waiting for src bucket item count to get to: 200000 curr count: 110225
2021/12/11 13:38:51 Waiting for src bucket item count to get to: 200000 curr count: 110225
2021/12/11 13:38:56 Waiting for src bucket item count to get to: 200000 curr count: 127174
2021/12/11 13:39:01 Waiting for src bucket item count to get to: 200000 curr count: 127174
2021/12/11 13:39:06 Waiting for src bucket item count to get to: 200000 curr count: 144234
2021/12/11 13:39:11 Waiting for src bucket item count to get to: 200000 curr count: 144234
2021/12/11 13:39:16 Waiting for src bucket item count to get to: 200000 curr count: 161044
2021/12/11 13:39:21 Waiting for src bucket item count to get to: 200000 curr count: 161044
2021/12/11 13:39:26 Waiting for src bucket item count to get to: 200000 curr count: 177865
2021/12/11 13:39:31 Waiting for src bucket item count to get to: 200000 curr count: 177865
2021/12/11 13:39:36 Waiting for src bucket item count to get to: 200000 curr count: 194920
2021/12/11 13:39:41 Waiting for src bucket item count to get to: 200000 curr count: 194920
2021/12/11 13:39:51 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:39:57 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:02 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:07 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:12 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:17 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:22 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:27 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:32 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:37 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/11 13:40:37 Post SET ops. Expected item count: 200000 got 200001
2021/12/11 13:40:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:40:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 13:40:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 13:40:52 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 13:40:57 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/11 13:40:57 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/11 13:40:57 No eventing-con process running
2021/12/11 13:40:57 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/11 13:40:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:41:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:41:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:41:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:41:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:41:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:41:20 Waiting for dst bucket item count to get to: 0 curr count: 36984
2021/12/11 13:41:25 Waiting for dst bucket item count to get to: 0 curr count: 36984
2021/12/11 13:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (287.59s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/11 13:41:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 13:41:31 Removing node(s): [] from the cluster
2021/12/11 13:41:36 Rebalance progress: 100
2021/12/11 13:41:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 13:41:41 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/11 13:41:41 No eventing-con process running
2021/12/11 13:41:41 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/11 13:41:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:41:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:41:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:41:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:41:59 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:42:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 13:42:05 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 13:42:10 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 13:42:15 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 13:42:20 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/11 13:42:20 App: TestDuplicateWithEventingRebOut got deployed
2021/12/11 13:42:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 13:42:20 Starting bucket ops to source bucket
2021/12/11 13:42:25 Rebalance progress: 25
2021/12/11 13:42:30 Rebalance progress: 25
2021/12/11 13:42:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:42:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:42:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:42:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 13:42:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:42:35 Waiting for src bucket item count to get to: 200000 curr count: 14415
2021/12/11 13:42:40 Waiting for src bucket item count to get to: 200000 curr count: 31184
2021/12/11 13:42:45 Waiting for src bucket item count to get to: 200000 curr count: 31184
2021/12/11 13:42:50 Waiting for src bucket item count to get to: 200000 curr count: 48158
2021/12/11 13:42:55 Waiting for src bucket item count to get to: 200000 curr count: 48158
2021/12/11 13:43:00 Waiting for src bucket item count to get to: 200000 curr count: 65190
2021/12/11 13:43:05 Waiting for src bucket item count to get to: 200000 curr count: 65190
2021/12/11 13:43:10 Waiting for src bucket item count to get to: 200000 curr count: 65190
2021/12/11 13:43:15 Waiting for src bucket item count to get to: 200000 curr count: 81646
2021/12/11 13:43:20 Waiting for src bucket item count to get to: 200000 curr count: 81646
2021/12/11 13:43:25 Waiting for src bucket item count to get to: 200000 curr count: 98357
2021/12/11 13:43:30 Waiting for src bucket item count to get to: 200000 curr count: 98357
2021/12/11 13:43:35 Waiting for src bucket item count to get to: 200000 curr count: 115538
2021/12/11 13:43:40 Waiting for src bucket item count to get to: 200000 curr count: 115538
2021/12/11 13:43:45 Waiting for src bucket item count to get to: 200000 curr count: 132443
2021/12/11 13:43:50 Waiting for src bucket item count to get to: 200000 curr count: 132443
2021/12/11 13:43:55 Waiting for src bucket item count to get to: 200000 curr count: 149307
2021/12/11 13:44:00 Waiting for src bucket item count to get to: 200000 curr count: 149307
2021/12/11 13:44:05 Waiting for src bucket item count to get to: 200000 curr count: 166289
2021/12/11 13:44:11 Waiting for src bucket item count to get to: 200000 curr count: 166289
2021/12/11 13:44:16 Waiting for src bucket item count to get to: 200000 curr count: 182694
2021/12/11 13:44:21 Waiting for src bucket item count to get to: 200000 curr count: 182694
2021/12/11 13:44:26 Waiting for src bucket item count to get to: 200000 curr count: 199115
2021/12/11 13:44:31 Waiting for src bucket item count to get to: 200000 curr count: 199115
2021/12/11 13:44:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 13:44:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 13:44:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:44:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 13:44:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 13:44:51 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 13:44:56 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/11 13:44:56 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/11 13:44:56 No eventing-con process running
2021/12/11 13:44:56 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/11 13:44:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:45:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:45:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:45:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:45:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:45:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:45:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (234.04s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/11 13:45:29 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 13:45:29 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/11 13:45:29 No eventing-con process running
2021/12/11 13:45:29 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/11 13:45:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:45:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:45:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:45:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:45:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:45:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:45:48 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:45:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 13:45:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 13:45:59 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 13:46:04 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 13:46:09 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/11 13:46:09 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/11 13:46:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 13:46:09 Starting bucket ops to source bucket
2021/12/11 13:46:09 Removing node(s): [] from the cluster
2021/12/11 13:46:14 Rebalance progress: 25
2021/12/11 13:46:19 Rebalance progress: 25
2021/12/11 13:46:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:46:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:46:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:46:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/11 13:46:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:46:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 13:46:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 13:46:37 Rebalance progress: 16.66666666666667
2021/12/11 13:46:42 Rebalance progress: 16.66666666666667
2021/12/11 13:46:47 Rebalance progress: 16.66666666666667
2021/12/11 13:46:52 Rebalance progress: 16.66666666666667
2021/12/11 13:46:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:46:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:46:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:46:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:46:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 13:47:02 Rebalance progress: 16.66666666666667
2021/12/11 13:47:07 Rebalance progress: 16.66666666666667
2021/12/11 13:47:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:47:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:47:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:47:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:47:12 Waiting for src bucket item count to get to: 200000 curr count: 76881
2021/12/11 13:47:17 Waiting for src bucket item count to get to: 200000 curr count: 93353
2021/12/11 13:47:22 Waiting for src bucket item count to get to: 200000 curr count: 93353
2021/12/11 13:47:27 Waiting for src bucket item count to get to: 200000 curr count: 110027
2021/12/11 13:47:32 Waiting for src bucket item count to get to: 200000 curr count: 110027
2021/12/11 13:47:37 Waiting for src bucket item count to get to: 200000 curr count: 126964
2021/12/11 13:47:42 Waiting for src bucket item count to get to: 200000 curr count: 126964
2021/12/11 13:47:47 Waiting for src bucket item count to get to: 200000 curr count: 143655
2021/12/11 13:47:52 Waiting for src bucket item count to get to: 200000 curr count: 160125
2021/12/11 13:47:57 Waiting for src bucket item count to get to: 200000 curr count: 160125
2021/12/11 13:48:02 Waiting for src bucket item count to get to: 200000 curr count: 176169
2021/12/11 13:48:07 Waiting for src bucket item count to get to: 200000 curr count: 176169
2021/12/11 13:48:12 Waiting for src bucket item count to get to: 200000 curr count: 192478
2021/12/11 13:48:17 Waiting for src bucket item count to get to: 200000 curr count: 192478
2021/12/11 13:48:22 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 13:48:22 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 13:48:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:48:27 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 13:48:32 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 13:48:37 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 13:48:42 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/11 13:48:42 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/11 13:48:42 No eventing-con process running
2021/12/11 13:48:42 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/11 13:48:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:48:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:48:54 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:48:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:49:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:49:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:49:11 Waiting for dst bucket item count to get to: 0 curr count: 118560
2021/12/11 13:49:16 Waiting for dst bucket item count to get to: 0 curr count: 118560
2021/12/11 13:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (236.40s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/11 13:49:26 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 13:49:26 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/11 13:49:26 No eventing-con process running
2021/12/11 13:49:26 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/11 13:49:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:49:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:49:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:49:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:49:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:49:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:49:44 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:49:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 13:49:50 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 13:49:55 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 13:50:00 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 13:50:05 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/11 13:50:05 App: TestDuplicateWithKVRebIn got deployed
2021/12/11 13:50:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 13:50:05 Starting bucket ops to source bucket
2021/12/11 13:50:06 Removing node(s): [] from the cluster
2021/12/11 13:50:11 Rebalance progress: 16.66666666666667
2021/12/11 13:50:16 Rebalance progress: 16.66666666666667
2021/12/11 13:50:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:50:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:50:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:50:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:50:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 13:50:26 Rebalance progress: 16.66666666666667
2021/12/11 13:50:31 Rebalance progress: 16.66666666666667
2021/12/11 13:50:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:50:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:50:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:50:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:50:36 Waiting for src bucket item count to get to: 200000 curr count: 38120
2021/12/11 13:50:41 Waiting for src bucket item count to get to: 200000 curr count: 54227
2021/12/11 13:50:46 Waiting for src bucket item count to get to: 200000 curr count: 54227
2021/12/11 13:50:51 Waiting for src bucket item count to get to: 200000 curr count: 71035
2021/12/11 13:50:56 Waiting for src bucket item count to get to: 200000 curr count: 71035
2021/12/11 13:51:01 Waiting for src bucket item count to get to: 200000 curr count: 86815
2021/12/11 13:51:06 Waiting for src bucket item count to get to: 200000 curr count: 86815
2021/12/11 13:51:11 Waiting for src bucket item count to get to: 200000 curr count: 103434
2021/12/11 13:51:16 Waiting for src bucket item count to get to: 200000 curr count: 103434
2021/12/11 13:51:21 Waiting for src bucket item count to get to: 200000 curr count: 120169
2021/12/11 13:51:26 Waiting for src bucket item count to get to: 200000 curr count: 120169
2021/12/11 13:51:31 Waiting for src bucket item count to get to: 200000 curr count: 136965
2021/12/11 13:51:36 Waiting for src bucket item count to get to: 200000 curr count: 136965
2021/12/11 13:51:41 Waiting for src bucket item count to get to: 200000 curr count: 136965
2021/12/11 13:51:46 Waiting for src bucket item count to get to: 200000 curr count: 153566
2021/12/11 13:51:51 Waiting for src bucket item count to get to: 200000 curr count: 153566
2021/12/11 13:51:56 Waiting for src bucket item count to get to: 200000 curr count: 170359
2021/12/11 13:52:01 Waiting for src bucket item count to get to: 200000 curr count: 170359
2021/12/11 13:52:06 Waiting for src bucket item count to get to: 200000 curr count: 185955
2021/12/11 13:52:11 Waiting for src bucket item count to get to: 200000 curr count: 185955
2021/12/11 13:52:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 13:52:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 13:52:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:52:21 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 13:52:26 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 13:52:31 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 13:52:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/11 13:52:36 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/11 13:52:36 No eventing-con process running
2021/12/11 13:52:36 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/11 13:52:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:52:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:52:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:52:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (219.17s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/11 13:53:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 13:53:00 Removing node(s): [] from the cluster
2021/12/11 13:53:05 Rebalance progress: 100
2021/12/11 13:53:10 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 13:53:10 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/11 13:53:10 No eventing-con process running
2021/12/11 13:53:10 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/11 13:53:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:53:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:53:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:53:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:53:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:53:29 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:53:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 13:53:35 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 13:53:40 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 13:53:45 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 13:53:50 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/11 13:53:50 App: TestDuplicateWithKVRebOut got deployed
2021/12/11 13:53:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 13:53:50 Starting bucket ops to source bucket
2021/12/11 13:53:55 Rebalance progress: 16.66666666666667
2021/12/11 13:54:00 Rebalance progress: 16.66666666666667
2021/12/11 13:54:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:54:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:54:05 Waiting for src bucket item count to get to: 200000 curr count: 15145
2021/12/11 13:54:10 Waiting for src bucket item count to get to: 200000 curr count: 15145
2021/12/11 13:54:15 Waiting for src bucket item count to get to: 200000 curr count: 31595
2021/12/11 13:54:20 Waiting for src bucket item count to get to: 200000 curr count: 48391
2021/12/11 13:54:25 Waiting for src bucket item count to get to: 200000 curr count: 48391
2021/12/11 13:54:30 Waiting for src bucket item count to get to: 200000 curr count: 65049
2021/12/11 13:54:35 Waiting for src bucket item count to get to: 200000 curr count: 65049
2021/12/11 13:54:40 Waiting for src bucket item count to get to: 200000 curr count: 65049
2021/12/11 13:54:45 Waiting for src bucket item count to get to: 200000 curr count: 81080
2021/12/11 13:54:50 Waiting for src bucket item count to get to: 200000 curr count: 81080
2021/12/11 13:54:55 Waiting for src bucket item count to get to: 200000 curr count: 97226
2021/12/11 13:55:00 Waiting for src bucket item count to get to: 200000 curr count: 97226
2021/12/11 13:55:05 Waiting for src bucket item count to get to: 200000 curr count: 113730
2021/12/11 13:55:10 Waiting for src bucket item count to get to: 200000 curr count: 113730
2021/12/11 13:55:15 Waiting for src bucket item count to get to: 200000 curr count: 130128
2021/12/11 13:55:20 Waiting for src bucket item count to get to: 200000 curr count: 130128
2021/12/11 13:55:25 Waiting for src bucket item count to get to: 200000 curr count: 147061
2021/12/11 13:55:30 Waiting for src bucket item count to get to: 200000 curr count: 147061
2021/12/11 13:55:35 Waiting for src bucket item count to get to: 200000 curr count: 163470
2021/12/11 13:55:40 Waiting for src bucket item count to get to: 200000 curr count: 163470
2021/12/11 13:55:45 Waiting for src bucket item count to get to: 200000 curr count: 179543
2021/12/11 13:55:50 Waiting for src bucket item count to get to: 200000 curr count: 179543
2021/12/11 13:55:55 Waiting for src bucket item count to get to: 200000 curr count: 195527
2021/12/11 13:56:00 Waiting for src bucket item count to get to: 200000 curr count: 195527
2021/12/11 13:56:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 13:56:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 13:56:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:56:10 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 13:56:15 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 13:56:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 13:56:25 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/11 13:56:25 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/11 13:56:25 No eventing-con process running
2021/12/11 13:56:25 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/11 13:56:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:56:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:56:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:56:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:56:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:56:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 13:56:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (234.01s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/11 13:56:59 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 13:56:59 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/11 13:56:59 No eventing-con process running
2021/12/11 13:56:59 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/11 13:57:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:57:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 13:57:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:57:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:57:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:57:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 13:57:17 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 13:57:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 13:57:23 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 13:57:28 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 13:57:33 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 13:57:38 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/11 13:57:38 App: TestDuplicateWithKVSwapReb got deployed
2021/12/11 13:57:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 13:57:38 Starting bucket ops to source bucket
2021/12/11 13:57:39 Removing node(s): [] from the cluster
2021/12/11 13:57:44 Rebalance progress: 16.66666666666667
2021/12/11 13:57:49 Rebalance progress: 16.66666666666667
2021/12/11 13:57:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:57:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:57:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 13:57:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 13:57:59 Rebalance progress: 16.66666666666667
2021/12/11 13:58:04 Rebalance progress: 16.66666666666667
2021/12/11 13:58:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:58:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:58:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 13:58:14 Rebalance progress: 16.66666666666667
2021/12/11 13:58:19 Rebalance progress: 16.66666666666667
2021/12/11 13:58:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:58:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 13:58:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 13:58:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 13:58:24 Waiting for src bucket item count to get to: 200000 curr count: 65198
2021/12/11 13:58:29 Waiting for src bucket item count to get to: 200000 curr count: 65198
2021/12/11 13:58:34 Waiting for src bucket item count to get to: 200000 curr count: 80827
2021/12/11 13:58:39 Waiting for src bucket item count to get to: 200000 curr count: 80827
2021/12/11 13:58:44 Waiting for src bucket item count to get to: 200000 curr count: 97074
2021/12/11 13:58:49 Waiting for src bucket item count to get to: 200000 curr count: 97074
2021/12/11 13:58:54 Waiting for src bucket item count to get to: 200000 curr count: 113840
2021/12/11 13:58:59 Waiting for src bucket item count to get to: 200000 curr count: 113840
2021/12/11 13:59:04 Waiting for src bucket item count to get to: 200000 curr count: 130543
2021/12/11 13:59:09 Waiting for src bucket item count to get to: 200000 curr count: 130543
2021/12/11 13:59:14 Waiting for src bucket item count to get to: 200000 curr count: 130543
2021/12/11 13:59:19 Waiting for src bucket item count to get to: 200000 curr count: 147165
2021/12/11 13:59:24 Waiting for src bucket item count to get to: 200000 curr count: 147165
2021/12/11 13:59:29 Waiting for src bucket item count to get to: 200000 curr count: 163816
2021/12/11 13:59:34 Waiting for src bucket item count to get to: 200000 curr count: 179629
2021/12/11 13:59:39 Waiting for src bucket item count to get to: 200000 curr count: 179629
2021/12/11 13:59:44 Waiting for src bucket item count to get to: 200000 curr count: 195657
2021/12/11 13:59:49 Waiting for src bucket item count to get to: 200000 curr count: 195657
2021/12/11 13:59:54 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 13:59:54 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 13:59:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 13:59:59 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 14:00:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 14:00:10 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 14:00:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/11 14:00:15 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/11 14:00:15 No eventing-con process running
2021/12/11 14:00:15 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/11 14:00:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:00:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:00:26 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 14:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:00:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:00:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:00:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/11 14:00:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.40s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/11 14:00:48 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:00:48 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 14:00:48 No eventing-con process running
2021/12/11 14:00:48 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/11 14:00:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:00:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:01:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:01:07 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:01:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/11 14:01:13 Starting bucket ops to source bucket
2021/12/11 14:01:18 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:01:23 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:01:28 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:01:28 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:01:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 14:01:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:01:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:01:34 Rebalance progress: 16.66666666666667
2021/12/11 14:01:39 Rebalance progress: 16.66666666666667
2021/12/11 14:01:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:01:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 14:01:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:01:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:01:49 Rebalance progress: 16.66666666666667
2021/12/11 14:01:54 Rebalance progress: 16.66666666666667
2021/12/11 14:01:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:01:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:01:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/11 14:01:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 14:02:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:02:09 Rebalance progress: 12.5
2021/12/11 14:02:14 Rebalance progress: 12.5
2021/12/11 14:02:19 Rebalance progress: 12.5
2021/12/11 14:02:24 Rebalance progress: 88.70967741935483
2021/12/11 14:02:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:02:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:02:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:02:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:02:34 Rebalance progress: 12.5
2021/12/11 14:02:39 Rebalance progress: 12.5
2021/12/11 14:02:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:02:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:02:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:02:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:02:50 Rebalance progress: 12.5
2021/12/11 14:02:55 Rebalance progress: 12.5
2021/12/11 14:03:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:03:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:03:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:03:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 14:03:05 Rebalance progress: 12.5
2021/12/11 14:03:10 Rebalance progress: 12.5
2021/12/11 14:03:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:03:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:03:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:03:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:03:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:03:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:03:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:03:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:03:35 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 14:03:35 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:03:35 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/11 14:03:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:03:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:03:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12185
2021/12/11 14:03:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:03:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:03:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:03:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12185
2021/12/11 14:03:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.30s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/11 14:04:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 14:04:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/11 14:04:08 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:04:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/11 14:04:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:04:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:04:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:04:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:04:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:04:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:04:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:04:27 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:04:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/11 14:04:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:04:38 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/11 14:04:43 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/11 14:04:48 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/11 14:04:48 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:04:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:04:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:04:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:04:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:04:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 14:04:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:04:54 Rebalance progress: 12.5
2021/12/11 14:04:59 Rebalance progress: 12.5
2021/12/11 14:05:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:05:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 14:05:09 Rebalance progress: 12.5
2021/12/11 14:05:14 Rebalance progress: 12.5
2021/12/11 14:05:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:05:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:05:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:05:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:05:24 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 14:05:29 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 14:05:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 14:05:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/11 14:05:39 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/11 14:05:39 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:05:39 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/11 14:05:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:05:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:05:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4575
2021/12/11 14:05:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4575
2021/12/11 14:05:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:05:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:06:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:06:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4575
2021/12/11 14:06:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.39s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/11 14:06:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 14:06:18 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 14:06:18 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:06:18 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/11 14:06:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:06:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:06:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:06:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:06:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:06:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:06:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:06:36 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:06:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/11 14:06:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:06:48 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/11 14:06:53 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/11 14:06:58 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/11 14:06:58 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:06:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:06:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:06:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:06:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:07:03 Rebalance progress: 12.5
2021/12/11 14:07:08 Rebalance progress: 12.5
2021/12/11 14:07:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:07:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:07:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:07:18 Rebalance progress: 12.5
2021/12/11 14:07:23 Rebalance progress: 12.5
2021/12/11 14:07:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:07:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 14:07:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:07:34 Rebalance progress: 12.5
2021/12/11 14:07:39 Rebalance progress: 12.5
2021/12/11 14:07:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:07:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:07:49 Rebalance progress: 12.5
2021/12/11 14:07:54 Rebalance progress: 12.5
2021/12/11 14:07:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:07:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:07:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:08:04 Rebalance progress: 12.5
2021/12/11 14:08:09 Rebalance progress: 12.5
2021/12/11 14:08:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:08:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:08:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:08:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 14:08:19 Rebalance progress: 12.5
2021/12/11 14:08:24 Rebalance progress: 12.5
2021/12/11 14:08:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:08:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:08:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:08:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:08:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 14:08:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 14:08:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 14:08:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/11 14:08:49 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/11 14:08:49 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:08:49 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/11 14:08:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:08:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:09:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10657
2021/12/11 14:09:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10657
2021/12/11 14:09:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:09:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:09:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:09:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10654
2021/12/11 14:09:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.16s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/11 14:09:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 14:09:28 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/11 14:09:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:09:28 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/11 14:09:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:09:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:09:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:09:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:09:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:09:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:09:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:09:42 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:09:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 14:09:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:09:53 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/11 14:09:58 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/11 14:10:03 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/11 14:10:03 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:10:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:10:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:10:08 Rebalance progress: 12.5
2021/12/11 14:10:13 Rebalance progress: 12.5
2021/12/11 14:10:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:10:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:10:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:10:24 Rebalance progress: 12.5
2021/12/11 14:10:29 Rebalance progress: 12.5
2021/12/11 14:10:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:10:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:10:39 Rebalance progress: 12.5
2021/12/11 14:10:44 Rebalance progress: 12.5
2021/12/11 14:10:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:10:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:10:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:10:54 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 14:10:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 14:11:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 14:11:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/11 14:11:09 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/11 14:11:09 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:11:09 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/11 14:11:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:11:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:11:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6154
2021/12/11 14:11:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6154
2021/12/11 14:11:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:11:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:11:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:11:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6154
2021/12/11 14:11:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (140.01s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/11 14:11:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:11:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:11:48 Rebalance progress: 100
2021/12/11 14:11:53 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 14:11:54 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/11 14:11:54 No eventing-con process running
2021/12/11 14:11:54 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/11 14:11:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:12:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:12:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:12:07 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:12:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/11 14:12:13 Starting bucket ops to source bucket
2021/12/11 14:12:18 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/11 14:12:23 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/11 14:12:28 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/11 14:12:28 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:12:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:12:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:12:34 Rebalance progress: 12.5
2021/12/11 14:12:39 Rebalance progress: 12.5
2021/12/11 14:12:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:12:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 14:12:44 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/11 14:12:49 Rebalance progress: 12.5
2021/12/11 14:12:54 Rebalance progress: 12.5
2021/12/11 14:12:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:12:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:12:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:13:04 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 14:13:09 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 14:13:14 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 14:13:19 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/11 14:13:19 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/11 14:13:19 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:13:19 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/11 14:13:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:13:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:13:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4428
2021/12/11 14:13:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4428
2021/12/11 14:13:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:13:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:13:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:13:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4575
2021/12/11 14:13:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.67s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/11 14:13:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:13:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:13:58 Rebalance progress: 100
2021/12/11 14:14:03 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 14:14:03 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/11 14:14:03 No eventing-con process running
2021/12/11 14:14:03 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/11 14:14:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:14:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:14:17 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:14:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/11 14:14:23 Starting bucket ops to source bucket
2021/12/11 14:14:28 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/11 14:14:33 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/11 14:14:38 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/11 14:14:38 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:14:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:14:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:14:43 Rebalance progress: 12.5
2021/12/11 14:14:48 Rebalance progress: 12.5
2021/12/11 14:14:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:14:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:14:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 14:14:53 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/11 14:14:58 Rebalance progress: 12.5
2021/12/11 14:15:03 Rebalance progress: 12.5
2021/12/11 14:15:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:15:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:15:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:15:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:15:08 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/11 14:15:13 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 14:15:18 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 14:15:23 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 14:15:28 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/11 14:15:28 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/11 14:15:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:15:28 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/11 14:15:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:15:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:15:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11699
2021/12/11 14:15:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11699
2021/12/11 14:15:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:15:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:15:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:15:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37845
2021/12/11 14:15:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.36s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/11 14:16:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 14:16:07 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/11 14:16:07 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:16:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/11 14:16:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:16:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:16:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:16:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:16:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:16:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:16:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:16:26 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:16:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:16:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:16:27 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:16:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 14:16:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:16:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/11 14:16:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/11 14:16:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/11 14:16:48 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/11 14:16:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/11 14:16:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/11 14:17:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/11 14:17:03 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:17:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:17:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:17:08 Rebalance progress: 12.5
2021/12/11 14:17:13 Rebalance progress: 12.5
2021/12/11 14:17:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:17:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:17:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:17:24 Rebalance progress: 12.5
2021/12/11 14:17:29 Rebalance progress: 12.5
2021/12/11 14:17:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:17:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 14:17:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:17:39 Rebalance progress: 12.5
2021/12/11 14:17:44 Rebalance progress: 12.5
2021/12/11 14:17:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:17:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:17:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:17:54 Rebalance progress: 12.5
2021/12/11 14:17:59 Rebalance progress: 12.5
2021/12/11 14:18:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:18:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:18:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:18:10 Rebalance progress: 12.5
2021/12/11 14:18:15 Rebalance progress: 12.5
2021/12/11 14:18:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:18:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:18:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:18:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 14:18:25 Rebalance progress: 12.5
2021/12/11 14:18:30 Rebalance progress: 12.5
2021/12/11 14:18:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:18:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:18:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:18:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:18:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 14:18:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 14:18:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 14:18:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/11 14:18:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/11 14:18:55 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:18:55 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/11 14:18:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:19:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:19:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12185
2021/12/11 14:19:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:19:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:19:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:19:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1490
2021/12/11 14:19:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1490
2021/12/11 14:19:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:19:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:19:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 14:19:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 14:19:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 14:19:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/11 14:19:45 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/11 14:19:45 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:19:45 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/11 14:19:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:19:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:19:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:19:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:19:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:19:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (241.78s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/11 14:20:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 14:20:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/11 14:20:09 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:20:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/11 14:20:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:20:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:20:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:20:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:20:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:20:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:20:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:20:27 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:20:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:20:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:20:28 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:20:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 14:20:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:20:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/11 14:20:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/11 14:20:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/11 14:20:50 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/11 14:20:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/11 14:21:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/11 14:21:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/11 14:21:05 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:21:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:21:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:21:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/11 14:21:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:21:11 Rebalance progress: 12.5
2021/12/11 14:21:16 Rebalance progress: 12.5
2021/12/11 14:21:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:21:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:21:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:21:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 14:21:26 Rebalance progress: 12.5
2021/12/11 14:21:31 Rebalance progress: 12.5
2021/12/11 14:21:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:21:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:21:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:21:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:21:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 14:21:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 14:21:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 14:21:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/11 14:21:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/11 14:21:56 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:21:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/11 14:21:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:22:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:22:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6121
2021/12/11 14:22:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:22:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:22:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:22:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6121
2021/12/11 14:22:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6121
2021/12/11 14:22:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:22:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:22:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 14:22:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 14:22:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 14:22:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/11 14:22:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/11 14:22:46 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:22:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/11 14:22:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:22:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:22:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:22:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.07s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/11 14:23:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 14:23:10 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/11 14:23:10 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:23:10 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/11 14:23:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:23:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:23:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 14:23:28 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/11 14:23:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:23:28 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/11 14:23:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:23:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:42 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:23:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:23:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:23:43 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:23:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 14:23:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:23:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/11 14:23:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/11 14:24:04 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/11 14:24:04 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/11 14:24:09 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/11 14:24:14 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/11 14:24:19 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/11 14:24:19 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:24:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:24:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:24:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:24:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:24:25 Rebalance progress: 12.5
2021/12/11 14:24:30 Rebalance progress: 12.5
2021/12/11 14:24:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:24:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:24:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:24:40 Rebalance progress: 12.5
2021/12/11 14:24:45 Rebalance progress: 12.5
2021/12/11 14:24:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:24:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:24:55 Rebalance progress: 12.5
2021/12/11 14:25:00 Rebalance progress: 12.5
2021/12/11 14:25:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:25:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:25:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 14:25:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 14:25:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 14:25:26 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/11 14:25:26 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/11 14:25:26 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:25:26 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/11 14:25:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:25:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:25:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7619
2021/12/11 14:25:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:25:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:25:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:25:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6602
2021/12/11 14:25:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6602
2021/12/11 14:25:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:25:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:26:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 14:26:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 14:26:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 14:26:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/11 14:26:16 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/11 14:26:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:26:16 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/11 14:26:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:26:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:26:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:26:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:26:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:26:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.66s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 14:26:40 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:26:40 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 14:26:40 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:26:40 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/11 14:26:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:26:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:26:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:26:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:26:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:26:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:26:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:26:58 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:26:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/11 14:27:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:27:09 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:27:14 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:27:19 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:27:19 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:27:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:27:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:27:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:27:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:27:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:27:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:27:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:28:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:28:16 Rebalance progress: 12.5
2021/12/11 14:28:21 Rebalance progress: 12.5
2021/12/11 14:28:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:28:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:28:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:28:31 Rebalance progress: 12.5
2021/12/11 14:28:36 Rebalance progress: 12.5
2021/12/11 14:28:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:28:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:28:46 Rebalance progress: 12.5
2021/12/11 14:28:51 Rebalance progress: 12.5
2021/12/11 14:28:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:28:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:29:01 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:29:06 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:29:11 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:29:16 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:29:16 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 14:29:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:29:16 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/11 14:29:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:29:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:29:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11191
2021/12/11 14:29:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:29:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:29:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:29:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.31s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 14:29:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:29:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 14:29:45 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:29:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/11 14:29:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:29:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:29:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:29:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:29:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:29:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:30:03 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:30:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/11 14:30:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:30:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:30:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:30:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/11 14:30:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:30:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:30:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:30:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:30:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:30:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:30:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:31:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:31:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:31:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:31:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:31:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:31:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:31:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:32:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:32:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:32:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:32:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:32:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:32:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:33:04 Rebalance progress: 12.5
2021/12/11 14:33:09 Rebalance progress: 12.5
2021/12/11 14:33:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:33:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:33:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:33:20 Rebalance progress: 12.5
2021/12/11 14:33:25 Rebalance progress: 12.5
2021/12/11 14:33:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:33:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:33:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:33:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:33:35 Rebalance progress: 12.5
2021/12/11 14:33:40 Rebalance progress: 12.5
2021/12/11 14:33:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:33:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:33:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:33:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:33:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:33:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:34:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:34:05 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/11 14:34:05 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/11 14:34:05 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:34:05 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/11 14:34:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:34:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:34:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21551
2021/12/11 14:34:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:34:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:34:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:34:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21551
2021/12/11 14:34:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (294.05s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/11 14:34:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 14:34:39 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/11 14:34:39 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:34:39 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/11 14:34:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:34:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:34:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:34:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:34:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:34:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:34:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:34:57 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:34:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 14:35:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:35:09 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/11 14:35:14 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/11 14:35:19 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/11 14:35:19 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:35:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:35:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:35:24 Rebalance progress: 12.5
2021/12/11 14:35:29 Rebalance progress: 12.5
2021/12/11 14:35:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:35:34 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:35:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:35:49 Rebalance progress: 12.5
2021/12/11 14:35:54 Rebalance progress: 12.5
2021/12/11 14:35:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:35:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:35:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:36:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:36:05 Rebalance progress: 12.5
2021/12/11 14:36:10 Rebalance progress: 12.5
2021/12/11 14:36:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:36:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:36:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:36:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:36:20 Rebalance progress: 12.5
2021/12/11 14:36:25 Rebalance progress: 12.5
2021/12/11 14:36:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:36:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:36:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:36:30 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:36:35 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 14:36:40 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 14:36:45 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 14:36:50 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/11 14:36:50 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/11 14:36:50 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:36:50 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/11 14:36:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:36:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:37:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8600
2021/12/11 14:37:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8600
2021/12/11 14:37:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:37:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:37:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:37:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8600
2021/12/11 14:37:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.54s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/11 14:37:28 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 14:37:28 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/11 14:37:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:37:28 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/11 14:37:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:37:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:37:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:37:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:37:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:37:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:37:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:37:47 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:37:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/11 14:37:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:37:58 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/11 14:38:03 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/11 14:38:08 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/11 14:38:08 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:38:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/11 14:38:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:38:14 Rebalance progress: 12.5
2021/12/11 14:38:19 Rebalance progress: 12.5
2021/12/11 14:38:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:38:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/11 14:38:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:38:29 Rebalance progress: 12.5
2021/12/11 14:38:34 Rebalance progress: 12.5
2021/12/11 14:38:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:38:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:38:44 Rebalance progress: 12.5
2021/12/11 14:38:49 Rebalance progress: 12.5
2021/12/11 14:38:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:38:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:38:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:38:59 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 14:39:04 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 14:39:09 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 14:39:14 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/11 14:39:14 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/11 14:39:14 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:39:15 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/11 14:39:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:39:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:39:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6108
2021/12/11 14:39:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:39:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:39:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:39:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6108
2021/12/11 14:39:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.57s)
=== RUN   TestEventingRebInWithOps
2021/12/11 14:39:48 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 14:39:48 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/11 14:39:48 eventingSetup.go:46: No eventing-con process running
2021/12/11 14:39:48 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/11 14:39:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:39:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:39:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:39:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:40:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:40:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:40:07 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:40:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 14:40:13 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 14:40:18 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 14:40:23 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 14:40:28 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/11 14:40:28 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/11 14:40:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:40:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 14:40:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 14:40:33 Rebalance progress: 12.5
2021/12/11 14:40:38 Rebalance progress: 12.5
2021/12/11 14:40:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:40:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:40:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:40:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:40:49 Rebalance progress: 12.5
2021/12/11 14:40:54 Rebalance progress: 12.5
2021/12/11 14:40:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:40:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:40:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:40:59 Waiting for src bucket item count to get to: 200000 curr count: 33223
2021/12/11 14:41:04 Waiting for src bucket item count to get to: 200000 curr count: 49103
2021/12/11 14:41:09 Waiting for src bucket item count to get to: 200000 curr count: 49103
2021/12/11 14:41:14 Waiting for src bucket item count to get to: 200000 curr count: 65627
2021/12/11 14:41:19 Waiting for src bucket item count to get to: 200000 curr count: 65627
2021/12/11 14:41:24 Waiting for src bucket item count to get to: 200000 curr count: 81109
2021/12/11 14:41:29 Waiting for src bucket item count to get to: 200000 curr count: 81109
2021/12/11 14:41:34 Waiting for src bucket item count to get to: 200000 curr count: 97404
2021/12/11 14:41:39 Waiting for src bucket item count to get to: 200000 curr count: 97404
2021/12/11 14:41:44 Waiting for src bucket item count to get to: 200000 curr count: 113948
2021/12/11 14:41:49 Waiting for src bucket item count to get to: 200000 curr count: 113948
2021/12/11 14:41:54 Waiting for src bucket item count to get to: 200000 curr count: 130471
2021/12/11 14:41:59 Waiting for src bucket item count to get to: 200000 curr count: 130471
2021/12/11 14:42:04 Waiting for src bucket item count to get to: 200000 curr count: 146944
2021/12/11 14:42:09 Waiting for src bucket item count to get to: 200000 curr count: 146944
2021/12/11 14:42:14 Waiting for src bucket item count to get to: 200000 curr count: 146944
2021/12/11 14:42:19 Waiting for src bucket item count to get to: 200000 curr count: 163225
2021/12/11 14:42:24 Waiting for src bucket item count to get to: 200000 curr count: 163225
2021/12/11 14:42:29 Waiting for src bucket item count to get to: 200000 curr count: 178548
2021/12/11 14:42:34 Waiting for src bucket item count to get to: 200000 curr count: 178548
2021/12/11 14:42:39 Waiting for src bucket item count to get to: 200000 curr count: 194720
2021/12/11 14:42:44 Waiting for src bucket item count to get to: 200000 curr count: 194720
2021/12/11 14:42:49 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 14:42:49 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 14:42:49 Starting bucket ops to source bucket
2021/12/11 14:46:34 Waiting for dst bucket item count to get to: 0 curr count: 18019
2021/12/11 14:46:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:46:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 14:46:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:46:39 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 14:46:44 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 14:46:49 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 14:46:54 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/11 14:46:54 App: TestEventingRebInWithOps got un-deployed
2021/12/11 14:46:54 No eventing-con process running
2021/12/11 14:46:54 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/11 14:46:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:47:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:47:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:47:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:47:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:47:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.80s)
=== RUN   TestEventingRebOutWithOps
2021/12/11 14:47:13 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:47:13 Removing node(s): [] from the cluster
2021/12/11 14:47:18 Rebalance progress: 100
2021/12/11 14:47:23 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 14:47:23 App: TestEventingRebOutWithOps got un-deployed
2021/12/11 14:47:23 No eventing-con process running
2021/12/11 14:47:23 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/11 14:47:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:47:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:47:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:47:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:47:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:47:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:47:42 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:47:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 14:47:48 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 14:47:53 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 14:47:58 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 14:48:03 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/11 14:48:03 App: TestEventingRebOutWithOps got deployed
2021/12/11 14:48:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:48:03 Starting bucket ops to source bucket
2021/12/11 14:48:08 Rebalance progress: 12.5
2021/12/11 14:48:13 Rebalance progress: 12.5
2021/12/11 14:48:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:48:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:48:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:48:18 Waiting for src bucket item count to get to: 200000 curr count: 9591
2021/12/11 14:48:23 Waiting for src bucket item count to get to: 200000 curr count: 25761
2021/12/11 14:48:28 Waiting for src bucket item count to get to: 200000 curr count: 25761
2021/12/11 14:48:33 Waiting for src bucket item count to get to: 200000 curr count: 42243
2021/12/11 14:48:38 Waiting for src bucket item count to get to: 200000 curr count: 42243
2021/12/11 14:48:43 Waiting for src bucket item count to get to: 200000 curr count: 42243
2021/12/11 14:48:48 Waiting for src bucket item count to get to: 200000 curr count: 58699
2021/12/11 14:48:53 Waiting for src bucket item count to get to: 200000 curr count: 58699
2021/12/11 14:48:58 Waiting for src bucket item count to get to: 200000 curr count: 74677
2021/12/11 14:49:03 Waiting for src bucket item count to get to: 200000 curr count: 74677
2021/12/11 14:49:08 Waiting for src bucket item count to get to: 200000 curr count: 90785
2021/12/11 14:49:13 Waiting for src bucket item count to get to: 200000 curr count: 90785
2021/12/11 14:49:18 Waiting for src bucket item count to get to: 200000 curr count: 107307
2021/12/11 14:49:23 Waiting for src bucket item count to get to: 200000 curr count: 107307
2021/12/11 14:49:28 Waiting for src bucket item count to get to: 200000 curr count: 123746
2021/12/11 14:49:33 Waiting for src bucket item count to get to: 200000 curr count: 123746
2021/12/11 14:49:38 Waiting for src bucket item count to get to: 200000 curr count: 140417
2021/12/11 14:49:43 Waiting for src bucket item count to get to: 200000 curr count: 140417
2021/12/11 14:49:48 Waiting for src bucket item count to get to: 200000 curr count: 157019
2021/12/11 14:49:53 Waiting for src bucket item count to get to: 200000 curr count: 157019
2021/12/11 14:49:58 Waiting for src bucket item count to get to: 200000 curr count: 172768
2021/12/11 14:50:03 Waiting for src bucket item count to get to: 200000 curr count: 172768
2021/12/11 14:50:08 Waiting for src bucket item count to get to: 200000 curr count: 188599
2021/12/11 14:50:13 Waiting for src bucket item count to get to: 200000 curr count: 188599
2021/12/11 14:50:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 14:50:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 14:50:18 Starting bucket ops to source bucket
2021/12/11 14:54:03 Waiting for dst bucket item count to get to: 0 curr count: 15856
2021/12/11 14:54:08 Waiting for dst bucket item count to get to: 0 curr count: 15856
2021/12/11 14:54:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:54:08 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 14:54:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 14:54:13 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 14:54:18 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 14:54:23 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 14:54:28 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/11 14:54:28 App: TestEventingRebOutWithOps got un-deployed
2021/12/11 14:54:28 No eventing-con process running
2021/12/11 14:54:28 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/11 14:54:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:54:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:54:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:54:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:54:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (454.00s)
=== RUN   TestEventingSwapRebWithOps
2021/12/11 14:54:52 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 14:54:52 App: TestEventingSwapRebWithOps got un-deployed
2021/12/11 14:54:52 No eventing-con process running
2021/12/11 14:54:52 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/11 14:54:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:54:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:54:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 14:55:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:55:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:55:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 14:55:10 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 14:55:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 14:55:16 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 14:55:21 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 14:55:26 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 14:55:31 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/11 14:55:32 App: TestEventingSwapRebWithOps got deployed
2021/12/11 14:55:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 14:55:32 Starting bucket ops to source bucket
2021/12/11 14:55:32 Removing node(s): [] from the cluster
2021/12/11 14:55:37 Rebalance progress: 12.5
2021/12/11 14:55:42 Rebalance progress: 12.5
2021/12/11 14:55:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:55:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 14:55:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 14:55:52 Rebalance progress: 12.5
2021/12/11 14:55:57 Rebalance progress: 12.5
2021/12/11 14:56:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:56:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 14:56:07 Rebalance progress: 12.5
2021/12/11 14:56:12 Rebalance progress: 12.5
2021/12/11 14:56:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 14:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 14:56:17 Waiting for src bucket item count to get to: 200000 curr count: 57626
2021/12/11 14:56:22 Waiting for src bucket item count to get to: 200000 curr count: 72937
2021/12/11 14:56:27 Waiting for src bucket item count to get to: 200000 curr count: 72937
2021/12/11 14:56:32 Waiting for src bucket item count to get to: 200000 curr count: 89009
2021/12/11 14:56:37 Waiting for src bucket item count to get to: 200000 curr count: 89009
2021/12/11 14:56:42 Waiting for src bucket item count to get to: 200000 curr count: 105505
2021/12/11 14:56:47 Waiting for src bucket item count to get to: 200000 curr count: 105505
2021/12/11 14:56:52 Waiting for src bucket item count to get to: 200000 curr count: 121581
2021/12/11 14:56:57 Waiting for src bucket item count to get to: 200000 curr count: 121581
2021/12/11 14:57:02 Waiting for src bucket item count to get to: 200000 curr count: 137736
2021/12/11 14:57:07 Waiting for src bucket item count to get to: 200000 curr count: 137736
2021/12/11 14:57:12 Waiting for src bucket item count to get to: 200000 curr count: 153788
2021/12/11 14:57:18 Waiting for src bucket item count to get to: 200000 curr count: 153788
2021/12/11 14:57:23 Waiting for src bucket item count to get to: 200000 curr count: 153788
2021/12/11 14:57:28 Waiting for src bucket item count to get to: 200000 curr count: 169033
2021/12/11 14:57:33 Waiting for src bucket item count to get to: 200000 curr count: 169033
2021/12/11 14:57:38 Waiting for src bucket item count to get to: 200000 curr count: 184873
2021/12/11 14:57:43 Waiting for src bucket item count to get to: 200000 curr count: 184873
2021/12/11 14:57:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 14:57:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 14:57:48 Starting bucket ops to source bucket
2021/12/11 15:01:32 Waiting for dst bucket item count to get to: 0 curr count: 15102
2021/12/11 15:01:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:01:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:01:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:01:38 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 15:01:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 15:01:48 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 15:01:53 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/11 15:01:53 App: TestEventingSwapRebWithOps got un-deployed
2021/12/11 15:01:53 No eventing-con process running
2021/12/11 15:01:53 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/11 15:01:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:01:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:01:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:02:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (444.41s)
=== RUN   TestKVRebInWithOps
2021/12/11 15:02:16 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 15:02:16 App: TestKVRebInWithOps got un-deployed
2021/12/11 15:02:16 No eventing-con process running
2021/12/11 15:02:16 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/11 15:02:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:02:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:02:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:02:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:02:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:02:35 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:02:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 15:02:41 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 15:02:46 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 15:02:51 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 15:02:56 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/11 15:02:56 App: TestKVRebInWithOps got deployed
2021/12/11 15:02:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 15:02:56 Starting bucket ops to source bucket
2021/12/11 15:02:56 Removing node(s): [] from the cluster
2021/12/11 15:03:01 Rebalance progress: 12.5
2021/12/11 15:03:06 Rebalance progress: 12.5
2021/12/11 15:03:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:03:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:03:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:03:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:03:16 Rebalance progress: 12.5
2021/12/11 15:03:21 Rebalance progress: 12.5
2021/12/11 15:03:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:03:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:03:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:03:26 Waiting for src bucket item count to get to: 200000 curr count: 35939
2021/12/11 15:03:31 Waiting for src bucket item count to get to: 200000 curr count: 52147
2021/12/11 15:03:36 Waiting for src bucket item count to get to: 200000 curr count: 52147
2021/12/11 15:03:41 Waiting for src bucket item count to get to: 200000 curr count: 68337
2021/12/11 15:03:46 Waiting for src bucket item count to get to: 200000 curr count: 68337
2021/12/11 15:03:51 Waiting for src bucket item count to get to: 200000 curr count: 83453
2021/12/11 15:03:56 Waiting for src bucket item count to get to: 200000 curr count: 83453
2021/12/11 15:04:01 Waiting for src bucket item count to get to: 200000 curr count: 99887
2021/12/11 15:04:06 Waiting for src bucket item count to get to: 200000 curr count: 99887
2021/12/11 15:04:11 Waiting for src bucket item count to get to: 200000 curr count: 116256
2021/12/11 15:04:17 Waiting for src bucket item count to get to: 200000 curr count: 116256
2021/12/11 15:04:22 Waiting for src bucket item count to get to: 200000 curr count: 116256
2021/12/11 15:04:27 Waiting for src bucket item count to get to: 200000 curr count: 132557
2021/12/11 15:04:32 Waiting for src bucket item count to get to: 200000 curr count: 132557
2021/12/11 15:04:37 Waiting for src bucket item count to get to: 200000 curr count: 148735
2021/12/11 15:04:42 Waiting for src bucket item count to get to: 200000 curr count: 148735
2021/12/11 15:04:47 Waiting for src bucket item count to get to: 200000 curr count: 165048
2021/12/11 15:04:52 Waiting for src bucket item count to get to: 200000 curr count: 165048
2021/12/11 15:04:57 Waiting for src bucket item count to get to: 200000 curr count: 179873
2021/12/11 15:05:02 Waiting for src bucket item count to get to: 200000 curr count: 179873
2021/12/11 15:05:07 Waiting for src bucket item count to get to: 200000 curr count: 196220
2021/12/11 15:05:12 Waiting for src bucket item count to get to: 200000 curr count: 196220
2021/12/11 15:05:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 15:05:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 15:05:17 Starting bucket ops to source bucket
2021/12/11 15:09:02 Waiting for dst bucket item count to get to: 0 curr count: 13623
2021/12/11 15:09:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:09:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:09:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:09:07 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 15:09:12 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 15:09:17 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 15:09:22 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/11 15:09:22 App: TestKVRebInWithOps got un-deployed
2021/12/11 15:09:22 No eventing-con process running
2021/12/11 15:09:22 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/11 15:09:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:09:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:09:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:09:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.07s)
=== RUN   TestKVRebOutWithOps
2021/12/11 15:09:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 15:09:41 Removing node(s): [] from the cluster
2021/12/11 15:09:46 Rebalance progress: 100
2021/12/11 15:09:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 15:09:51 App: TestKVRebOutWithOps got un-deployed
2021/12/11 15:09:51 No eventing-con process running
2021/12/11 15:09:51 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/11 15:09:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:09:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:09:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:09:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:10:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:10:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:10:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:10:09 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:10:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 15:10:15 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 15:10:20 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 15:10:25 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 15:10:30 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/11 15:10:30 App: TestKVRebOutWithOps got deployed
2021/12/11 15:10:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:10:30 Starting bucket ops to source bucket
2021/12/11 15:10:35 Rebalance progress: 12.5
2021/12/11 15:10:41 Rebalance progress: 12.5
2021/12/11 15:10:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:10:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:10:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:10:46 Waiting for src bucket item count to get to: 200000 curr count: 13593
2021/12/11 15:10:51 Waiting for src bucket item count to get to: 200000 curr count: 29448
2021/12/11 15:10:56 Waiting for src bucket item count to get to: 200000 curr count: 29448
2021/12/11 15:11:01 Waiting for src bucket item count to get to: 200000 curr count: 45924
2021/12/11 15:11:06 Waiting for src bucket item count to get to: 200000 curr count: 45924
2021/12/11 15:11:11 Waiting for src bucket item count to get to: 200000 curr count: 62201
2021/12/11 15:11:16 Waiting for src bucket item count to get to: 200000 curr count: 62201
2021/12/11 15:11:21 Waiting for src bucket item count to get to: 200000 curr count: 62201
2021/12/11 15:11:26 Waiting for src bucket item count to get to: 200000 curr count: 77313
2021/12/11 15:11:31 Waiting for src bucket item count to get to: 200000 curr count: 77313
2021/12/11 15:11:36 Waiting for src bucket item count to get to: 200000 curr count: 93584
2021/12/11 15:11:41 Waiting for src bucket item count to get to: 200000 curr count: 93584
2021/12/11 15:11:46 Waiting for src bucket item count to get to: 200000 curr count: 110007
2021/12/11 15:11:51 Waiting for src bucket item count to get to: 200000 curr count: 110007
2021/12/11 15:11:56 Waiting for src bucket item count to get to: 200000 curr count: 126394
2021/12/11 15:12:01 Waiting for src bucket item count to get to: 200000 curr count: 126394
2021/12/11 15:12:06 Waiting for src bucket item count to get to: 200000 curr count: 142861
2021/12/11 15:12:11 Waiting for src bucket item count to get to: 200000 curr count: 142861
2021/12/11 15:12:16 Waiting for src bucket item count to get to: 200000 curr count: 159207
2021/12/11 15:12:21 Waiting for src bucket item count to get to: 200000 curr count: 159207
2021/12/11 15:12:26 Waiting for src bucket item count to get to: 200000 curr count: 174665
2021/12/11 15:12:31 Waiting for src bucket item count to get to: 200000 curr count: 174665
2021/12/11 15:12:36 Waiting for src bucket item count to get to: 200000 curr count: 190957
2021/12/11 15:12:41 Waiting for src bucket item count to get to: 200000 curr count: 190957
2021/12/11 15:12:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 15:12:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 15:12:46 Starting bucket ops to source bucket
2021/12/11 15:16:31 Waiting for dst bucket item count to get to: 0 curr count: 11574
2021/12/11 15:16:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:16:31 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:16:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:16:36 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 15:16:41 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 15:16:46 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 15:16:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/11 15:16:51 App: TestKVRebOutWithOps got un-deployed
2021/12/11 15:16:51 No eventing-con process running
2021/12/11 15:16:51 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/11 15:16:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:16:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:16:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:16:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:17:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:17:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (449.13s)
=== RUN   TestKVSwapRebWithOps
2021/12/11 15:17:14 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 15:17:14 App: TestKVSwapRebWithOps got un-deployed
2021/12/11 15:17:14 No eventing-con process running
2021/12/11 15:17:14 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/11 15:17:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:17:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:17:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:17:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:17:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:17:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:17:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:17:33 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:17:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/11 15:17:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 15:17:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 15:17:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 15:17:54 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/11 15:17:54 App: TestKVSwapRebWithOps got deployed
2021/12/11 15:17:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 15:17:54 Starting bucket ops to source bucket
2021/12/11 15:17:55 Removing node(s): [] from the cluster
2021/12/11 15:18:00 Rebalance progress: 12.5
2021/12/11 15:18:05 Rebalance progress: 12.5
2021/12/11 15:18:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:18:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:18:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:18:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 15:18:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:18:15 Rebalance progress: 12.5
2021/12/11 15:18:20 Rebalance progress: 12.5
2021/12/11 15:18:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:18:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:18:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:18:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 15:18:30 Rebalance progress: 12.5
2021/12/11 15:18:35 Rebalance progress: 12.5
2021/12/11 15:18:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:18:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:18:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:18:41 Waiting for src bucket item count to get to: 200000 curr count: 54378
2021/12/11 15:18:46 Waiting for src bucket item count to get to: 200000 curr count: 70937
2021/12/11 15:18:51 Waiting for src bucket item count to get to: 200000 curr count: 70937
2021/12/11 15:18:56 Waiting for src bucket item count to get to: 200000 curr count: 85844
2021/12/11 15:19:01 Waiting for src bucket item count to get to: 200000 curr count: 85844
2021/12/11 15:19:06 Waiting for src bucket item count to get to: 200000 curr count: 102123
2021/12/11 15:19:11 Waiting for src bucket item count to get to: 200000 curr count: 118320
2021/12/11 15:19:16 Waiting for src bucket item count to get to: 200000 curr count: 118320
2021/12/11 15:19:21 Waiting for src bucket item count to get to: 200000 curr count: 134569
2021/12/11 15:19:26 Waiting for src bucket item count to get to: 200000 curr count: 134569
2021/12/11 15:19:31 Waiting for src bucket item count to get to: 200000 curr count: 150963
2021/12/11 15:19:36 Waiting for src bucket item count to get to: 200000 curr count: 150963
2021/12/11 15:19:41 Waiting for src bucket item count to get to: 200000 curr count: 167181
2021/12/11 15:19:46 Waiting for src bucket item count to get to: 200000 curr count: 167181
2021/12/11 15:19:51 Waiting for src bucket item count to get to: 200000 curr count: 182101
2021/12/11 15:19:56 Waiting for src bucket item count to get to: 200000 curr count: 182101
2021/12/11 15:20:01 Waiting for src bucket item count to get to: 200000 curr count: 198457
2021/12/11 15:20:06 Waiting for src bucket item count to get to: 200000 curr count: 198457
2021/12/11 15:20:11 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/11 15:20:11 Post SET ops. Expected item count: 200000 got 200000
2021/12/11 15:20:11 Starting bucket ops to source bucket
2021/12/11 15:23:56 Waiting for dst bucket item count to get to: 0 curr count: 21079
2021/12/11 15:24:01 Waiting for dst bucket item count to get to: 0 curr count: 3475
2021/12/11 15:24:06 Waiting for dst bucket item count to get to: 0 curr count: 3475
2021/12/11 15:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:24:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:24:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:24:11 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 15:24:16 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 15:24:21 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 15:24:26 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/11 15:24:26 App: TestKVSwapRebWithOps got un-deployed
2021/12/11 15:24:26 No eventing-con process running
2021/12/11 15:24:26 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/11 15:24:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:24:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:24:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:24:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:24:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:24:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (455.53s)
=== RUN   TestEventingRebInWithTimers
2021/12/11 15:24:50 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 15:24:50 App: TestEventingRebInWithTimers got un-deployed
2021/12/11 15:24:50 No eventing-con process running
2021/12/11 15:24:50 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/11 15:24:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:24:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:24:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:25:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:25:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:25:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:25:08 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:25:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 15:25:15 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 15:25:20 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 15:25:25 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 15:25:30 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/11 15:25:30 App: TestEventingRebInWithTimers got deployed
2021/12/11 15:25:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 15:25:30 Starting bucket ops to source bucket
2021/12/11 15:25:30 Removing node(s): [] from the cluster
2021/12/11 15:25:35 Rebalance progress: 12.5
2021/12/11 15:25:40 Rebalance progress: 12.5
2021/12/11 15:25:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:25:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:25:50 Rebalance progress: 12.5
2021/12/11 15:25:55 Rebalance progress: 12.5
2021/12/11 15:26:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:26:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:26:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:26:00 Waiting for src bucket item count to get to: 50000 curr count: 27181
2021/12/11 15:26:05 Waiting for src bucket item count to get to: 50000 curr count: 37325
2021/12/11 15:26:10 Waiting for src bucket item count to get to: 50000 curr count: 48621
2021/12/11 15:26:15 Waiting for src bucket item count to get to: 50000 curr count: 48621
2021/12/11 15:26:25 Waiting for dst bucket item count to get to: 50000 curr count: 41372
2021/12/11 15:26:30 Waiting for dst bucket item count to get to: 50000 curr count: 41372
2021/12/11 15:26:30 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:26:30 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:26:30 Starting bucket ops to source bucket
2021/12/11 15:27:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:27:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:27:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:27:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:28:00 Waiting for dst bucket item count to get to: 0 curr count: 41567
2021/12/11 15:28:05 Waiting for dst bucket item count to get to: 0 curr count: 41567
2021/12/11 15:28:10 Waiting for dst bucket item count to get to: 0 curr count: 23683
2021/12/11 15:28:15 Waiting for dst bucket item count to get to: 0 curr count: 23683
2021/12/11 15:28:20 Waiting for dst bucket item count to get to: 0 curr count: 12485
2021/12/11 15:28:25 Waiting for dst bucket item count to get to: 0 curr count: 12485
2021/12/11 15:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:28:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:28:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:28:30 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 15:28:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 15:28:40 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 15:28:45 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/11 15:28:45 App: TestEventingRebInWithTimers got un-deployed
2021/12/11 15:28:45 No eventing-con process running
2021/12/11 15:28:45 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/11 15:28:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:28:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:28:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:28:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:28:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (258.48s)
=== RUN   TestEventingRebOutWithTimers
2021/12/11 15:29:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 15:29:04 Removing node(s): [] from the cluster
2021/12/11 15:29:09 Rebalance progress: 100
2021/12/11 15:29:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 15:29:14 App: TestEventingRebOutWithTimers got un-deployed
2021/12/11 15:29:14 No eventing-con process running
2021/12/11 15:29:14 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/11 15:29:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:29:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:29:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:29:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:29:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:29:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:29:32 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:29:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 15:29:38 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 15:29:43 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 15:29:48 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 15:29:53 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/11 15:29:53 App: TestEventingRebOutWithTimers got deployed
2021/12/11 15:29:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:29:53 Starting bucket ops to source bucket
2021/12/11 15:29:59 Rebalance progress: 12.5
2021/12/11 15:30:04 Rebalance progress: 12.5
2021/12/11 15:30:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:30:09 Waiting for src bucket item count to get to: 50000 curr count: 7722
2021/12/11 15:30:14 Waiting for src bucket item count to get to: 50000 curr count: 21303
2021/12/11 15:30:19 Waiting for src bucket item count to get to: 50000 curr count: 21303
2021/12/11 15:30:24 Waiting for src bucket item count to get to: 50000 curr count: 33133
2021/12/11 15:30:29 Waiting for src bucket item count to get to: 50000 curr count: 33133
2021/12/11 15:30:34 Waiting for src bucket item count to get to: 50000 curr count: 33133
2021/12/11 15:30:39 Waiting for src bucket item count to get to: 50000 curr count: 44189
2021/12/11 15:30:44 Waiting for src bucket item count to get to: 50000 curr count: 44189
2021/12/11 15:30:54 Waiting for dst bucket item count to get to: 50000 curr count: 38126
2021/12/11 15:30:59 Waiting for dst bucket item count to get to: 50000 curr count: 38126
2021/12/11 15:31:04 Waiting for dst bucket item count to get to: 50000 curr count: 48724
2021/12/11 15:31:09 Waiting for dst bucket item count to get to: 50000 curr count: 48724
2021/12/11 15:31:09 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:31:09 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:31:09 Starting bucket ops to source bucket
2021/12/11 15:32:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:32:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:32:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:32:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:32:38 Waiting for dst bucket item count to get to: 0 curr count: 34510
2021/12/11 15:32:43 Waiting for dst bucket item count to get to: 0 curr count: 34510
2021/12/11 15:32:48 Waiting for dst bucket item count to get to: 0 curr count: 22802
2021/12/11 15:32:53 Waiting for dst bucket item count to get to: 0 curr count: 22802
2021/12/11 15:32:58 Waiting for dst bucket item count to get to: 0 curr count: 3085
2021/12/11 15:33:03 Waiting for dst bucket item count to get to: 0 curr count: 3085
2021/12/11 15:33:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:33:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:33:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:33:08 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 15:33:13 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 15:33:18 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 15:33:23 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/11 15:33:23 App: TestEventingRebOutWithTimers got un-deployed
2021/12/11 15:33:23 No eventing-con process running
2021/12/11 15:33:23 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/11 15:33:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:33:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:33:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:33:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:33:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:33:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (278.11s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/11 15:33:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 15:33:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/11 15:33:47 No eventing-con process running
2021/12/11 15:33:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/11 15:33:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:33:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:33:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:34:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:34:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:34:05 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:34:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 15:34:11 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 15:34:16 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 15:34:21 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 15:34:26 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/11 15:34:26 App: TestEventingSwapRebWithTimers got deployed
2021/12/11 15:34:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 15:34:26 Starting bucket ops to source bucket
2021/12/11 15:34:27 Removing node(s): [] from the cluster
2021/12/11 15:34:32 Rebalance progress: 12.5
2021/12/11 15:34:37 Rebalance progress: 12.5
2021/12/11 15:34:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:34:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:34:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:34:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 15:34:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:34:47 Rebalance progress: 12.5
2021/12/11 15:34:52 Rebalance progress: 12.5
2021/12/11 15:34:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:34:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:34:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 15:35:03 Rebalance progress: 12.5
2021/12/11 15:35:08 Rebalance progress: 12.5
2021/12/11 15:35:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:35:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:35:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:35:18 Waiting for dst bucket item count to get to: 50000 curr count: 35801
2021/12/11 15:35:23 Waiting for dst bucket item count to get to: 50000 curr count: 35801
2021/12/11 15:35:28 Waiting for dst bucket item count to get to: 50000 curr count: 44756
2021/12/11 15:35:33 Waiting for dst bucket item count to get to: 50000 curr count: 44756
2021/12/11 15:35:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:35:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:35:33 Starting bucket ops to source bucket
2021/12/11 15:36:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:36:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:36:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:36:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:37:02 Waiting for dst bucket item count to get to: 0 curr count: 40639
2021/12/11 15:37:07 Waiting for dst bucket item count to get to: 0 curr count: 40639
2021/12/11 15:37:12 Waiting for dst bucket item count to get to: 0 curr count: 30890
2021/12/11 15:37:17 Waiting for dst bucket item count to get to: 0 curr count: 30890
2021/12/11 15:37:22 Waiting for dst bucket item count to get to: 0 curr count: 6151
2021/12/11 15:37:27 Waiting for dst bucket item count to get to: 0 curr count: 6151
2021/12/11 15:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:37:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:37:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 15:37:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 15:37:42 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 15:37:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/11 15:37:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/11 15:37:47 No eventing-con process running
2021/12/11 15:37:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/11 15:37:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:37:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:37:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (263.91s)
=== RUN   TestKVRebInWithTimers
2021/12/11 15:38:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 15:38:10 App: TestKVRebInWithTimers got un-deployed
2021/12/11 15:38:10 No eventing-con process running
2021/12/11 15:38:10 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/11 15:38:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:38:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:38:29 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:38:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 15:38:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 15:38:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 15:38:45 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 15:38:50 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/11 15:38:50 App: TestKVRebInWithTimers got deployed
2021/12/11 15:38:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 15:38:50 Starting bucket ops to source bucket
2021/12/11 15:38:50 Removing node(s): [] from the cluster
2021/12/11 15:38:55 Rebalance progress: 12.5
2021/12/11 15:39:00 Rebalance progress: 12.5
2021/12/11 15:39:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:39:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:39:11 Rebalance progress: 12.5
2021/12/11 15:39:16 Rebalance progress: 12.5
2021/12/11 15:39:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:39:21 Waiting for src bucket item count to get to: 50000 curr count: 34868
2021/12/11 15:39:26 Waiting for src bucket item count to get to: 50000 curr count: 34868
2021/12/11 15:39:31 Waiting for src bucket item count to get to: 50000 curr count: 46801
2021/12/11 15:39:36 Waiting for src bucket item count to get to: 50000 curr count: 46801
2021/12/11 15:39:46 Waiting for dst bucket item count to get to: 50000 curr count: 43854
2021/12/11 15:39:51 Waiting for dst bucket item count to get to: 50000 curr count: 43854
2021/12/11 15:39:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:39:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:39:51 Starting bucket ops to source bucket
2021/12/11 15:41:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:41:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:41:10 Waiting for dst bucket item count to get to: 0 curr count: 49985
2021/12/11 15:41:15 Waiting for dst bucket item count to get to: 0 curr count: 49985
2021/12/11 15:41:20 Waiting for dst bucket item count to get to: 0 curr count: 41434
2021/12/11 15:41:25 Waiting for dst bucket item count to get to: 0 curr count: 41434
2021/12/11 15:41:30 Waiting for dst bucket item count to get to: 0 curr count: 16379
2021/12/11 15:41:35 Waiting for dst bucket item count to get to: 0 curr count: 16379
2021/12/11 15:41:40 Waiting for dst bucket item count to get to: 0 curr count: 3907
2021/12/11 15:41:45 Waiting for dst bucket item count to get to: 0 curr count: 3907
2021/12/11 15:41:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:41:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:41:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:41:50 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 15:41:55 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 15:42:00 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 15:42:05 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/11 15:42:05 App: TestKVRebInWithTimers got un-deployed
2021/12/11 15:42:05 No eventing-con process running
2021/12/11 15:42:05 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/11 15:42:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:42:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:42:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:42:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:42:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.14s)
=== RUN   TestKVRebOutWithTimers
2021/12/11 15:42:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 15:42:24 Removing node(s): [] from the cluster
2021/12/11 15:42:29 Rebalance progress: 100
2021/12/11 15:42:34 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 15:42:34 App: TestKVRebOutWithTimers got un-deployed
2021/12/11 15:42:34 No eventing-con process running
2021/12/11 15:42:34 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/11 15:42:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:42:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:42:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:42:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:42:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:42:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:42:53 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:42:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 15:42:59 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 15:43:04 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 15:43:09 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 15:43:14 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/11 15:43:14 App: TestKVRebOutWithTimers got deployed
2021/12/11 15:43:14 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:43:14 Starting bucket ops to source bucket
2021/12/11 15:43:19 Rebalance progress: 12.5
2021/12/11 15:43:24 Rebalance progress: 12.5
2021/12/11 15:43:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:43:29 Waiting for src bucket item count to get to: 50000 curr count: 7347
2021/12/11 15:43:34 Waiting for src bucket item count to get to: 50000 curr count: 21927
2021/12/11 15:43:39 Waiting for src bucket item count to get to: 50000 curr count: 21927
2021/12/11 15:43:44 Waiting for src bucket item count to get to: 50000 curr count: 34009
2021/12/11 15:43:49 Waiting for src bucket item count to get to: 50000 curr count: 34009
2021/12/11 15:43:54 Waiting for src bucket item count to get to: 50000 curr count: 45014
2021/12/11 15:43:59 Waiting for src bucket item count to get to: 50000 curr count: 45014
2021/12/11 15:44:04 Waiting for src bucket item count to get to: 50000 curr count: 45014
2021/12/11 15:44:14 Waiting for dst bucket item count to get to: 50000 curr count: 36634
2021/12/11 15:44:19 Waiting for dst bucket item count to get to: 50000 curr count: 36634
2021/12/11 15:44:24 Waiting for dst bucket item count to get to: 50000 curr count: 45221
2021/12/11 15:44:29 Waiting for dst bucket item count to get to: 50000 curr count: 45221
2021/12/11 15:44:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:44:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:44:29 Starting bucket ops to source bucket
2021/12/11 15:45:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:45:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:45:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:45:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:45:59 Waiting for dst bucket item count to get to: 0 curr count: 36413
2021/12/11 15:46:04 Waiting for dst bucket item count to get to: 0 curr count: 36413
2021/12/11 15:46:09 Waiting for dst bucket item count to get to: 0 curr count: 25636
2021/12/11 15:46:14 Waiting for dst bucket item count to get to: 0 curr count: 25636
2021/12/11 15:46:19 Waiting for dst bucket item count to get to: 0 curr count: 9121
2021/12/11 15:46:24 Waiting for dst bucket item count to get to: 0 curr count: 9121
2021/12/11 15:46:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:46:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:46:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:46:29 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 15:46:34 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 15:46:39 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 15:46:44 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/11 15:46:44 App: TestKVRebOutWithTimers got un-deployed
2021/12/11 15:46:44 No eventing-con process running
2021/12/11 15:46:44 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/11 15:46:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:46:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:46:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:46:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:46:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:46:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.17s)
=== RUN   TestKVSwapRebWithTimers
2021/12/11 15:47:08 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:47:08 App: TestKVSwapRebWithTimers got un-deployed
2021/12/11 15:47:08 No eventing-con process running
2021/12/11 15:47:08 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/11 15:47:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:47:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:47:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:47:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:47:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:47:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:47:26 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:47:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/11 15:47:32 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 15:47:37 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 15:47:42 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 15:47:47 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/11 15:47:47 App: TestKVSwapRebWithTimers got deployed
2021/12/11 15:47:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 15:47:47 Starting bucket ops to source bucket
2021/12/11 15:47:48 Removing node(s): [] from the cluster
2021/12/11 15:47:53 Rebalance progress: 12.5
2021/12/11 15:47:58 Rebalance progress: 12.5
2021/12/11 15:48:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:48:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:48:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:48:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 15:48:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:48:09 Rebalance progress: 12.5
2021/12/11 15:48:14 Rebalance progress: 12.5
2021/12/11 15:48:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:48:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:48:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:48:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 15:48:24 Rebalance progress: 12.5
2021/12/11 15:48:29 Rebalance progress: 12.5
2021/12/11 15:48:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:48:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:48:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:48:34 Waiting for src bucket item count to get to: 50000 curr count: 48843
2021/12/11 15:48:39 Waiting for src bucket item count to get to: 50000 curr count: 48843
2021/12/11 15:48:49 Waiting for dst bucket item count to get to: 50000 curr count: 48476
2021/12/11 15:48:54 Waiting for dst bucket item count to get to: 50000 curr count: 48476
2021/12/11 15:48:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:48:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:48:54 Starting bucket ops to source bucket
2021/12/11 15:50:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:50:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:50:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:50:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:50:24 Waiting for dst bucket item count to get to: 0 curr count: 36438
2021/12/11 15:50:29 Waiting for dst bucket item count to get to: 0 curr count: 36438
2021/12/11 15:50:34 Waiting for dst bucket item count to get to: 0 curr count: 24619
2021/12/11 15:50:39 Waiting for dst bucket item count to get to: 0 curr count: 24619
2021/12/11 15:50:44 Waiting for dst bucket item count to get to: 0 curr count: 6232
2021/12/11 15:50:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:50:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/11 15:50:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:50:49 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:50:54 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:50:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:51:04 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:51:09 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:51:14 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/11 15:51:14 App: TestKVSwapRebWithTimers got un-deployed
2021/12/11 15:51:14 No eventing-con process running
2021/12/11 15:51:14 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/11 15:51:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:51:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:51:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:51:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:51:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (269.82s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/11 15:51:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:51:38 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/11 15:51:38 No eventing-con process running
2021/12/11 15:51:38 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/11 15:51:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:51:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:51:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:51:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:51:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:51:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:51:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:51:56 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:51:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 15:52:02 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 15:52:07 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 15:52:12 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 15:52:17 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/11 15:52:17 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/11 15:52:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 15:52:17 Starting bucket ops to source bucket
2021/12/11 15:52:18 Removing node(s): [] from the cluster
2021/12/11 15:52:23 Rebalance progress: 12.5
2021/12/11 15:52:28 Rebalance progress: 12.5
2021/12/11 15:52:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:52:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:52:38 Rebalance progress: 12.5
2021/12/11 15:52:43 Rebalance progress: 12.5
2021/12/11 15:52:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:52:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:52:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:52:48 Waiting for src bucket item count to get to: 50000 curr count: 28853
2021/12/11 15:52:53 Waiting for src bucket item count to get to: 50000 curr count: 40071
2021/12/11 15:52:58 Waiting for src bucket item count to get to: 50000 curr count: 40071
2021/12/11 15:53:08 Waiting for dst bucket item count to get to: 50000 curr count: 24816
2021/12/11 15:53:13 Waiting for dst bucket item count to get to: 50000 curr count: 24816
2021/12/11 15:53:18 Waiting for dst bucket item count to get to: 50000 curr count: 36500
2021/12/11 15:53:23 Waiting for dst bucket item count to get to: 50000 curr count: 36500
2021/12/11 15:53:28 Waiting for dst bucket item count to get to: 50000 curr count: 43364
2021/12/11 15:53:33 Waiting for dst bucket item count to get to: 50000 curr count: 43364
2021/12/11 15:53:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:53:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:53:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:53:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:53:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:53:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:53:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:53:58 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:54:03 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/11 15:54:03 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/11 15:54:03 No eventing-con process running
2021/12/11 15:54:03 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/11 15:54:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:54:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:54:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:54:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:54:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:54:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:54:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.33s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/11 15:54:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 15:54:32 Removing node(s): [] from the cluster
2021/12/11 15:54:38 Rebalance progress: 100
2021/12/11 15:54:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:54:43 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/11 15:54:43 No eventing-con process running
2021/12/11 15:54:43 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/11 15:54:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:54:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:54:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:54:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:54:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:55:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:55:01 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:55:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 15:55:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 15:55:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 15:55:17 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 15:55:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/11 15:55:22 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/11 15:55:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:55:22 Starting bucket ops to source bucket
2021/12/11 15:55:27 Rebalance progress: 12.5
2021/12/11 15:55:32 Rebalance progress: 12.5
2021/12/11 15:55:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:55:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:55:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:55:37 Waiting for src bucket item count to get to: 50000 curr count: 8976
2021/12/11 15:55:42 Waiting for src bucket item count to get to: 50000 curr count: 23050
2021/12/11 15:55:47 Waiting for src bucket item count to get to: 50000 curr count: 23050
2021/12/11 15:55:52 Waiting for src bucket item count to get to: 50000 curr count: 36157
2021/12/11 15:55:57 Waiting for src bucket item count to get to: 50000 curr count: 36157
2021/12/11 15:56:02 Waiting for src bucket item count to get to: 50000 curr count: 45478
2021/12/11 15:56:07 Waiting for src bucket item count to get to: 50000 curr count: 45478
2021/12/11 15:56:12 Waiting for src bucket item count to get to: 50000 curr count: 45478
2021/12/11 15:56:22 Waiting for dst bucket item count to get to: 50000 curr count: 31846
2021/12/11 15:56:27 Waiting for dst bucket item count to get to: 50000 curr count: 31846
2021/12/11 15:56:32 Waiting for dst bucket item count to get to: 50000 curr count: 46652
2021/12/11 15:56:37 Waiting for dst bucket item count to get to: 50000 curr count: 46652
2021/12/11 15:56:37 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:56:37 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:56:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:56:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:56:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:56:53 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:56:58 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:57:03 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:57:08 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/11 15:57:08 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/11 15:57:08 No eventing-con process running
2021/12/11 15:57:08 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/11 15:57:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:57:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:57:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:57:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:57:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:57:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 15:57:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.31s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/11 15:57:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 15:57:36 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/11 15:57:36 No eventing-con process running
2021/12/11 15:57:36 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/11 15:57:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:57:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:57:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 15:57:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:57:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:57:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:57:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 15:57:55 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 15:57:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 15:58:01 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 15:58:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 15:58:11 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 15:58:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/11 15:58:16 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/11 15:58:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 15:58:16 Starting bucket ops to source bucket
2021/12/11 15:58:16 Removing node(s): [] from the cluster
2021/12/11 15:58:21 Rebalance progress: 12.5
2021/12/11 15:58:26 Rebalance progress: 12.5
2021/12/11 15:58:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:58:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:58:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:58:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 15:58:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 15:58:37 Rebalance progress: 12.5
2021/12/11 15:58:42 Rebalance progress: 12.5
2021/12/11 15:58:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:58:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:58:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:58:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 15:58:52 Rebalance progress: 12.5
2021/12/11 15:58:57 Rebalance progress: 12.5
2021/12/11 15:59:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:59:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 15:59:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 15:59:07 Waiting for dst bucket item count to get to: 50000 curr count: 27299
2021/12/11 15:59:12 Waiting for dst bucket item count to get to: 50000 curr count: 27299
2021/12/11 15:59:17 Waiting for dst bucket item count to get to: 50000 curr count: 36428
2021/12/11 15:59:22 Waiting for dst bucket item count to get to: 50000 curr count: 36428
2021/12/11 15:59:27 Waiting for dst bucket item count to get to: 50000 curr count: 46989
2021/12/11 15:59:32 Waiting for dst bucket item count to get to: 50000 curr count: 46989
2021/12/11 15:59:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 15:59:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 15:59:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 15:59:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 15:59:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 15:59:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 15:59:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 15:59:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 16:00:02 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/11 16:00:02 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/11 16:00:02 No eventing-con process running
2021/12/11 16:00:02 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/11 16:00:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:00:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:00:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:00:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:00:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:00:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:00:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:00:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (189.80s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/11 16:00:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:00:46 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/11 16:00:46 No eventing-con process running
2021/12/11 16:00:46 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/11 16:00:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:00:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:00:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:00:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:01:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:01:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:01:05 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:01:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 16:01:11 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 16:01:16 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 16:01:21 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 16:01:26 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/11 16:01:26 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/11 16:01:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:01:26 Starting bucket ops to source bucket
2021/12/11 16:01:26 Removing node(s): [] from the cluster
2021/12/11 16:01:31 Rebalance progress: 12.5
2021/12/11 16:01:36 Rebalance progress: 12.5
2021/12/11 16:01:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:01:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:01:46 Rebalance progress: 12.5
2021/12/11 16:01:51 Rebalance progress: 12.5
2021/12/11 16:01:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:01:56 Waiting for src bucket item count to get to: 50000 curr count: 30065
2021/12/11 16:02:01 Waiting for src bucket item count to get to: 50000 curr count: 41287
2021/12/11 16:02:06 Waiting for src bucket item count to get to: 50000 curr count: 41287
2021/12/11 16:02:16 Waiting for dst bucket item count to get to: 50000 curr count: 25715
2021/12/11 16:02:21 Waiting for dst bucket item count to get to: 50000 curr count: 25715
2021/12/11 16:02:26 Waiting for dst bucket item count to get to: 50000 curr count: 35187
2021/12/11 16:02:31 Waiting for dst bucket item count to get to: 50000 curr count: 35187
2021/12/11 16:02:36 Waiting for dst bucket item count to get to: 50000 curr count: 47589
2021/12/11 16:02:41 Waiting for dst bucket item count to get to: 50000 curr count: 47589
2021/12/11 16:02:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:02:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:02:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:02:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:02:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:02:57 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:03:02 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:03:07 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:03:12 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/11 16:03:12 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/11 16:03:12 No eventing-con process running
2021/12/11 16:03:12 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/11 16:03:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:03:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:03:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:03:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:03:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:03:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:03:40 Waiting for dst bucket item count to get to: 0 curr count: 20039
2021/12/11 16:03:45 Waiting for dst bucket item count to get to: 0 curr count: 20039
2021/12/11 16:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.34s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/11 16:03:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:03:51 Removing node(s): [] from the cluster
2021/12/11 16:03:56 Rebalance progress: 100
2021/12/11 16:04:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:04:01 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/11 16:04:01 No eventing-con process running
2021/12/11 16:04:01 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/11 16:04:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:04:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:04:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:04:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:04:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:04:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:04:19 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:04:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 16:04:26 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 16:04:31 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 16:04:36 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 16:04:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/11 16:04:41 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/11 16:04:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:04:41 Starting bucket ops to source bucket
2021/12/11 16:04:46 Rebalance progress: 12.5
2021/12/11 16:04:51 Rebalance progress: 12.5
2021/12/11 16:04:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:04:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:04:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:04:56 Waiting for src bucket item count to get to: 50000 curr count: 11515
2021/12/11 16:05:01 Waiting for src bucket item count to get to: 50000 curr count: 25244
2021/12/11 16:05:06 Waiting for src bucket item count to get to: 50000 curr count: 25244
2021/12/11 16:05:11 Waiting for src bucket item count to get to: 50000 curr count: 38530
2021/12/11 16:05:16 Waiting for src bucket item count to get to: 50000 curr count: 38530
2021/12/11 16:05:21 Waiting for src bucket item count to get to: 50000 curr count: 48541
2021/12/11 16:05:26 Waiting for src bucket item count to get to: 50000 curr count: 48541
2021/12/11 16:05:31 Waiting for src bucket item count to get to: 50000 curr count: 48541
2021/12/11 16:05:41 Waiting for dst bucket item count to get to: 50000 curr count: 31397
2021/12/11 16:05:46 Waiting for dst bucket item count to get to: 50000 curr count: 31397
2021/12/11 16:05:51 Waiting for dst bucket item count to get to: 50000 curr count: 45011
2021/12/11 16:05:56 Waiting for dst bucket item count to get to: 50000 curr count: 45011
2021/12/11 16:05:56 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:05:56 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:05:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:06:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:06:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:06:11 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:06:16 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:06:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:06:26 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/11 16:06:26 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/11 16:06:26 No eventing-con process running
2021/12/11 16:06:26 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/11 16:06:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:06:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:06:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:06:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:06:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:06:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:06:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.33s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/11 16:06:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:06:55 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/11 16:06:55 No eventing-con process running
2021/12/11 16:06:55 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/11 16:06:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:07:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:07:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:07:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:07:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:07:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:07:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:07:13 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:07:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 16:07:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 16:07:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 16:07:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 16:07:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/11 16:07:34 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/11 16:07:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:07:34 Starting bucket ops to source bucket
2021/12/11 16:07:35 Removing node(s): [] from the cluster
2021/12/11 16:07:40 Rebalance progress: 12.5
2021/12/11 16:07:45 Rebalance progress: 12.5
2021/12/11 16:07:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:07:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:07:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:07:56 Rebalance progress: 12.5
2021/12/11 16:08:01 Rebalance progress: 12.5
2021/12/11 16:08:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:08:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:08:11 Rebalance progress: 12.5
2021/12/11 16:08:16 Rebalance progress: 12.5
2021/12/11 16:08:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:08:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:08:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:08:21 Waiting for src bucket item count to get to: 50000 curr count: 42851
2021/12/11 16:08:31 Waiting for dst bucket item count to get to: 50000 curr count: 30235
2021/12/11 16:08:36 Waiting for dst bucket item count to get to: 50000 curr count: 36282
2021/12/11 16:08:41 Waiting for dst bucket item count to get to: 50000 curr count: 36282
2021/12/11 16:08:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:08:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:08:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:08:46 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:08:51 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:08:56 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:09:01 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:09:06 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:09:11 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/11 16:09:11 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/11 16:09:11 No eventing-con process running
2021/12/11 16:09:11 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/11 16:09:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:09:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:09:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:09:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (169.90s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/11 16:09:45 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:09:45 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/11 16:09:45 No eventing-con process running
2021/12/11 16:09:45 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/11 16:09:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:09:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:09:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:09:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:09:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:09:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:10:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:10:03 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:10:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 16:10:09 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 16:10:14 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 16:10:19 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 16:10:24 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/11 16:10:24 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/11 16:10:24 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 16:10:24 Starting bucket ops to source bucket
2021/12/11 16:10:25 Removing node(s): [] from the cluster
2021/12/11 16:10:30 Rebalance progress: 12.5
2021/12/11 16:10:35 Rebalance progress: 12.5
2021/12/11 16:10:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:10:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:10:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:10:45 Rebalance progress: 12.5
2021/12/11 16:10:50 Rebalance progress: 12.5
2021/12/11 16:10:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:10:55 Waiting for src bucket item count to get to: 50000 curr count: 30663
2021/12/11 16:11:00 Waiting for src bucket item count to get to: 50000 curr count: 30663
2021/12/11 16:11:05 Waiting for src bucket item count to get to: 50000 curr count: 41808
2021/12/11 16:11:10 Waiting for src bucket item count to get to: 50000 curr count: 41808
2021/12/11 16:11:20 Waiting for dst bucket item count to get to: 50000 curr count: 34122
2021/12/11 16:11:25 Waiting for dst bucket item count to get to: 50000 curr count: 41384
2021/12/11 16:11:30 Waiting for dst bucket item count to get to: 50000 curr count: 41384
2021/12/11 16:11:30 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:11:30 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:11:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:11:35 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:11:40 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:11:46 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:11:51 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:11:56 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:12:01 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/11 16:12:01 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/11 16:12:01 No eventing-con process running
2021/12/11 16:12:01 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/11 16:12:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:12:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:12:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:12:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:12:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:12:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:12:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:12:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:12:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (174.63s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/11 16:12:34 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 16:12:35 Removing node(s): [] from the cluster
2021/12/11 16:12:40 Rebalance progress: 100
2021/12/11 16:12:45 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:12:45 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/11 16:12:45 No eventing-con process running
2021/12/11 16:12:45 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/11 16:12:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:12:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:12:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:12:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:12:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:13:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:13:03 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:13:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 16:13:10 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 16:13:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 16:13:20 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 16:13:25 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/11 16:13:25 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/11 16:13:25 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:13:25 Starting bucket ops to source bucket
2021/12/11 16:13:30 Rebalance progress: 12.5
2021/12/11 16:13:35 Rebalance progress: 12.5
2021/12/11 16:13:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:13:40 Waiting for src bucket item count to get to: 50000 curr count: 5968
2021/12/11 16:13:45 Waiting for src bucket item count to get to: 50000 curr count: 19595
2021/12/11 16:13:50 Waiting for src bucket item count to get to: 50000 curr count: 19595
2021/12/11 16:13:55 Waiting for src bucket item count to get to: 50000 curr count: 31011
2021/12/11 16:14:00 Waiting for src bucket item count to get to: 50000 curr count: 42993
2021/12/11 16:14:05 Waiting for src bucket item count to get to: 50000 curr count: 42993
2021/12/11 16:14:10 Waiting for src bucket item count to get to: 50000 curr count: 42993
2021/12/11 16:14:20 Waiting for dst bucket item count to get to: 50000 curr count: 37463
2021/12/11 16:14:25 Waiting for dst bucket item count to get to: 50000 curr count: 37463
2021/12/11 16:14:30 Waiting for dst bucket item count to get to: 50000 curr count: 45997
2021/12/11 16:14:35 Waiting for dst bucket item count to get to: 50000 curr count: 45997
2021/12/11 16:14:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:14:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:14:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:14:40 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:14:45 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:14:50 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:14:55 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:15:00 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:15:05 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/11 16:15:05 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/11 16:15:05 No eventing-con process running
2021/12/11 16:15:05 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/11 16:15:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:15:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:15:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:15:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:15:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:15:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.60s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/11 16:15:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:15:39 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/11 16:15:39 No eventing-con process running
2021/12/11 16:15:39 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/11 16:15:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:15:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:15:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:15:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:15:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:15:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:15:57 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:15:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 16:16:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 16:16:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 16:16:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 16:16:19 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/11 16:16:19 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/11 16:16:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/11 16:16:19 Starting bucket ops to source bucket
2021/12/11 16:16:20 Removing node(s): [] from the cluster
2021/12/11 16:16:25 Rebalance progress: 12.5
2021/12/11 16:16:30 Rebalance progress: 12.5
2021/12/11 16:16:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:16:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 16:16:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:16:40 Rebalance progress: 12.5
2021/12/11 16:16:45 Rebalance progress: 12.5
2021/12/11 16:16:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:16:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:16:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:16:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:16:55 Rebalance progress: 12.5
2021/12/11 16:17:00 Rebalance progress: 12.5
2021/12/11 16:17:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:17:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:17:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:17:05 Waiting for src bucket item count to get to: 50000 curr count: 47529
2021/12/11 16:17:10 Waiting for src bucket item count to get to: 50000 curr count: 47529
2021/12/11 16:17:20 Waiting for dst bucket item count to get to: 50000 curr count: 44816
2021/12/11 16:17:25 Waiting for dst bucket item count to get to: 50000 curr count: 44816
2021/12/11 16:17:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:17:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:17:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:17:31 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:17:36 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:17:41 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:17:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:17:51 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:17:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/11 16:17:56 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/11 16:17:56 No eventing-con process running
2021/12/11 16:17:56 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/11 16:17:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:18:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:18:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:18:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:18:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.37s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/11 16:18:29 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 16:18:29 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/11 16:18:29 No eventing-con process running
2021/12/11 16:18:29 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/11 16:18:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:18:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:18:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:18:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:18:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:18:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:18:48 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:18:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/11 16:18:54 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 16:18:59 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 16:19:04 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 16:19:09 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/11 16:19:09 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/11 16:19:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:19:09 Starting bucket ops to source bucket
2021/12/11 16:19:09 Removing node(s): [] from the cluster
2021/12/11 16:19:14 Rebalance progress: 12.5
2021/12/11 16:19:20 Rebalance progress: 12.5
2021/12/11 16:19:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:19:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:19:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:19:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:19:30 Rebalance progress: 12.5
2021/12/11 16:19:35 Rebalance progress: 12.5
2021/12/11 16:19:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:19:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:19:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:19:40 Waiting for src bucket item count to get to: 50000 curr count: 27201
2021/12/11 16:19:45 Waiting for src bucket item count to get to: 50000 curr count: 36425
2021/12/11 16:19:50 Waiting for src bucket item count to get to: 50000 curr count: 36425
2021/12/11 16:19:55 Waiting for src bucket item count to get to: 50000 curr count: 48676
2021/12/11 16:20:00 Waiting for src bucket item count to get to: 50000 curr count: 48676
2021/12/11 16:20:10 Waiting for dst bucket item count to get to: 50000 curr count: 42418
2021/12/11 16:20:15 Waiting for dst bucket item count to get to: 50000 curr count: 42418
2021/12/11 16:20:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:20:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:20:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:20:20 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 16:20:25 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 16:20:30 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 16:20:35 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 16:20:40 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/11 16:20:40 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/11 16:20:40 No eventing-con process running
2021/12/11 16:20:40 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/11 16:20:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:20:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:20:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:20:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:20:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:20:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:21:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:21:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.48s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/11 16:21:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:21:14 Removing node(s): [] from the cluster
2021/12/11 16:21:19 Rebalance progress: 100
2021/12/11 16:21:24 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:21:24 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/11 16:21:24 No eventing-con process running
2021/12/11 16:21:24 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/11 16:21:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:21:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:21:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:21:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:21:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:21:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:21:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:21:42 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:21:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/11 16:21:49 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 16:21:54 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 16:21:59 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 16:22:04 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/11 16:22:04 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/11 16:22:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:22:04 Starting bucket ops to source bucket
2021/12/11 16:22:09 Rebalance progress: 12.5
2021/12/11 16:22:14 Rebalance progress: 12.5
2021/12/11 16:22:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:22:19 Waiting for src bucket item count to get to: 50000 curr count: 7295
2021/12/11 16:22:24 Waiting for src bucket item count to get to: 50000 curr count: 21069
2021/12/11 16:22:29 Waiting for src bucket item count to get to: 50000 curr count: 21069
2021/12/11 16:22:34 Waiting for src bucket item count to get to: 50000 curr count: 32453
2021/12/11 16:22:39 Waiting for src bucket item count to get to: 50000 curr count: 32453
2021/12/11 16:22:44 Waiting for src bucket item count to get to: 50000 curr count: 43194
2021/12/11 16:22:49 Waiting for src bucket item count to get to: 50000 curr count: 43194
2021/12/11 16:22:54 Waiting for src bucket item count to get to: 50000 curr count: 43194
2021/12/11 16:23:04 Waiting for dst bucket item count to get to: 50000 curr count: 34829
2021/12/11 16:23:09 Waiting for dst bucket item count to get to: 50000 curr count: 34829
2021/12/11 16:23:14 Waiting for dst bucket item count to get to: 50000 curr count: 45511
2021/12/11 16:23:19 Waiting for dst bucket item count to get to: 50000 curr count: 45511
2021/12/11 16:23:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:23:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:23:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:23:24 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:23:29 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:23:34 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:23:39 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:23:44 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:23:49 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/11 16:23:49 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/11 16:23:49 No eventing-con process running
2021/12/11 16:23:49 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/11 16:23:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:24:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:24:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:24:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:24:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:24:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:24:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (189.24s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/11 16:24:28 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:24:28 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/11 16:24:28 No eventing-con process running
2021/12/11 16:24:28 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/11 16:24:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:24:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:24:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:24:47 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:24:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/11 16:24:53 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 16:24:58 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 16:25:03 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 16:25:08 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/11 16:25:08 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/11 16:25:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:25:08 Starting bucket ops to source bucket
2021/12/11 16:25:08 Removing node(s): [] from the cluster
2021/12/11 16:25:13 Rebalance progress: 12.5
2021/12/11 16:25:18 Rebalance progress: 12.5
2021/12/11 16:25:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:25:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:25:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:25:29 Rebalance progress: 12.5
2021/12/11 16:25:34 Rebalance progress: 12.5
2021/12/11 16:25:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:25:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:25:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:25:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:25:44 Rebalance progress: 12.5
2021/12/11 16:25:49 Rebalance progress: 12.5
2021/12/11 16:25:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:25:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:25:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:25:54 Waiting for src bucket item count to get to: 50000 curr count: 47487
2021/12/11 16:25:59 Waiting for src bucket item count to get to: 50000 curr count: 47487
2021/12/11 16:26:09 Waiting for dst bucket item count to get to: 50000 curr count: 47209
2021/12/11 16:26:14 Waiting for dst bucket item count to get to: 50000 curr count: 47209
2021/12/11 16:26:14 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/11 16:26:14 Post SET ops. Expected item count: 50000 got 50000
2021/12/11 16:26:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:26:19 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:26:24 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:26:29 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:26:34 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:26:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:26:44 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/11 16:26:44 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/11 16:26:44 No eventing-con process running
2021/12/11 16:26:44 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/11 16:26:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:26:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:26:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:26:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:26:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:27:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:27:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/11 16:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.57s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/11 16:27:17 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:27:17 App: bucket_op_on_update got un-deployed
2021/12/11 16:27:17 No eventing-con process running
2021/12/11 16:27:17 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:27:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:27:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:27:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:27:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:27:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:27:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:27:36 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:27:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:27:42 Starting bucket ops to source bucket
2021/12/11 16:27:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:27:52 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:27:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:27:57 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:27:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:27:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:27:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:27:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:28:02 Rebalance progress: 12.5
2021/12/11 16:28:07 Rebalance progress: 12.5
2021/12/11 16:28:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:28:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:28:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:28:18 Rebalance progress: 12.5
2021/12/11 16:28:23 Rebalance progress: 12.5
2021/12/11 16:28:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:28:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 16:28:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:28:33 Rebalance progress: 12.5
2021/12/11 16:28:38 Rebalance progress: 12.5
2021/12/11 16:28:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:28:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:28:49 Rebalance progress: 12.5
2021/12/11 16:28:54 Rebalance progress: 12.5
2021/12/11 16:28:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:28:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:28:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:29:04 Rebalance progress: 12.5
2021/12/11 16:29:09 Rebalance progress: 12.5
2021/12/11 16:29:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:29:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:29:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:29:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 16:29:19 Rebalance progress: 12.5
2021/12/11 16:29:24 Rebalance progress: 12.5
2021/12/11 16:29:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:29:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:29:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:29:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:29:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:29:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:29:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:29:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:29:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:29:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:29:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:29:59 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:29:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:30:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:30:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:30:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10264
2021/12/11 16:30:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10264
2021/12/11 16:30:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:30:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:30:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:30:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10648
2021/12/11 16:30:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.24s)
=== RUN   TestKVForceFailover
2021/12/11 16:30:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:30:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:30:43 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:30:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:30:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:30:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:30:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:30:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:30:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:30:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:30:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:30:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:31:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:31:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:31:02 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:31:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:31:03 Starting bucket ops to source bucket
2021/12/11 16:31:08 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:31:13 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:31:18 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:31:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:31:23 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:31:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:31:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:31:23 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/11 16:31:43 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/11 16:31:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/11 16:31:49 Rebalance progress: 12.5
2021/12/11 16:31:54 Rebalance progress: 12.5
2021/12/11 16:31:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:31:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:31:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:31:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/11 16:32:04 Rebalance progress: 12.5
2021/12/11 16:32:09 Rebalance progress: 12.5
2021/12/11 16:32:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:32:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:32:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:32:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:32:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:32:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:32:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:32:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:32:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:32:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:32:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:32:44 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:32:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:32:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:32:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:32:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7067
2021/12/11 16:32:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:32:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:33:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:33:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7067
2021/12/11 16:33:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.99s)
=== RUN   TestKVHardFailoverRecovery
2021/12/11 16:33:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:33:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:33:18 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:33:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:33:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:33:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:33:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:33:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:33:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:33:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:33:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:33:36 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:33:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:33:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:33:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:33:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:33:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:33:57 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:33:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:33:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:33:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:33:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:34:03 Rebalance progress: 12.5
2021/12/11 16:34:08 Rebalance progress: 12.5
2021/12/11 16:34:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:34:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:34:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:34:13 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/11 16:35:13 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/11 16:35:18 Rebalance progress: 12.5
2021/12/11 16:35:23 Rebalance progress: 12.5
2021/12/11 16:35:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:35:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:35:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:35:33 Rebalance progress: 12.5
2021/12/11 16:35:38 Rebalance progress: 12.5
2021/12/11 16:35:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:35:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:35:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:35:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:35:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:35:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:35:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:36:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:36:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:36:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:36:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:36:13 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:36:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:36:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:36:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:36:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12035
2021/12/11 16:36:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:36:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:36:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:36:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12035
2021/12/11 16:36:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.26s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/11 16:36:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:36:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:36:47 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:36:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:36:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:36:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:36:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:36:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:37:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:37:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:37:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:37:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:37:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:37:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:37:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:37:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:37:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:37:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:37:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 16:37:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:37:33 Rebalance progress: 12.5
2021/12/11 16:37:38 Rebalance progress: 12.5
2021/12/11 16:37:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:37:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/11 16:37:48 Rebalance progress: 12.5
2021/12/11 16:37:53 Rebalance progress: 12.5
2021/12/11 16:37:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:37:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:38:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:38:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:38:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:38:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:38:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:38:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:38:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:38:28 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:38:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:38:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:38:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:38:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 465
2021/12/11 16:38:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 465
2021/12/11 16:38:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:38:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:38:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:38:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4593
2021/12/11 16:38:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.81s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/11 16:39:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:39:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:39:07 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:39:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:39:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:39:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:39:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:39:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:39:20 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:39:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:39:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:39:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:39:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:39:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:39:41 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:39:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:39:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:39:47 Rebalance progress: 12.5
2021/12/11 16:39:52 Rebalance progress: 12.5
2021/12/11 16:39:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:39:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:39:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:40:02 Rebalance progress: 12.5
2021/12/11 16:40:07 Rebalance progress: 12.5
2021/12/11 16:40:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:40:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:40:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:40:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:40:17 Rebalance progress: 12.5
2021/12/11 16:40:22 Rebalance progress: 12.5
2021/12/11 16:40:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:40:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:40:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:40:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:40:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:40:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:40:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:40:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:40:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:40:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:40:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:40:58 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:40:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:41:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:41:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:41:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:41:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:41:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:41:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6104
2021/12/11 16:41:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:41:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:41:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:41:21 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:41:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:41:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:41:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:41:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:41:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:41:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:41:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.20s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/11 16:41:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:41:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:41:45 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:41:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:41:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:41:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:41:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:41:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:41:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:41:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:42:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:42:04 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 16:42:04 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:42:04 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/11 16:42:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:42:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:42:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:42:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:42:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:42:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:42:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:42:17 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:42:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:42:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:42:18 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:42:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 16:42:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:42:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:42:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:42:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:42:39 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/11 16:42:44 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 16:42:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 16:42:54 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 16:42:54 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:42:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:42:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:42:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:42:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:43:00 Rebalance progress: 12.5
2021/12/11 16:43:05 Rebalance progress: 12.5
2021/12/11 16:43:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:43:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:43:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:43:15 Rebalance progress: 12.5
2021/12/11 16:43:20 Rebalance progress: 12.5
2021/12/11 16:43:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:43:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/11 16:43:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:43:30 Rebalance progress: 12.5
2021/12/11 16:43:35 Rebalance progress: 12.5
2021/12/11 16:43:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:43:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:43:46 Rebalance progress: 12.5
2021/12/11 16:43:51 Rebalance progress: 12.5
2021/12/11 16:43:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:43:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:43:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:44:01 Rebalance progress: 12.5
2021/12/11 16:44:06 Rebalance progress: 12.5
2021/12/11 16:44:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:44:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:44:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:44:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/11 16:44:16 Rebalance progress: 12.5
2021/12/11 16:44:21 Rebalance progress: 12.5
2021/12/11 16:44:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:44:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:44:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:44:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:44:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:44:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:44:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:44:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:44:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:44:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:45:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:45:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:45:01 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:45:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:45:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:45:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:45:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12130
2021/12/11 16:45:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12130
2021/12/11 16:45:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:45:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:45:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:45:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10525
2021/12/11 16:45:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10525
2021/12/11 16:45:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:45:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:45:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:45:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:45:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:45:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:46:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:46:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:46:06 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 16:46:06 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:46:06 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/11 16:46:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:46:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:46:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:46:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:46:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:46:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (284.99s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/11 16:46:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:46:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:46:30 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:46:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:46:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:46:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:46:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:46:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:46:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:46:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:46:49 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:46:49 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 16:46:49 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:46:49 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/11 16:46:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:46:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:46:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:46:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:47:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:47:02 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:47:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:47:03 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:47:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/11 16:47:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:47:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:47:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:47:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:47:24 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/11 16:47:29 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 16:47:34 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 16:47:39 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/11 16:47:39 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:47:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:47:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:47:45 Rebalance progress: 12.5
2021/12/11 16:47:50 Rebalance progress: 12.5
2021/12/11 16:47:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:47:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:47:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:47:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:48:00 Rebalance progress: 12.5
2021/12/11 16:48:05 Rebalance progress: 12.5
2021/12/11 16:48:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:48:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:48:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:48:16 Rebalance progress: 12.5
2021/12/11 16:48:21 Rebalance progress: 12.5
2021/12/11 16:48:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:48:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:48:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:48:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:48:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:48:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:48:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:48:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:48:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:48:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:48:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:48:56 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:48:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:49:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:49:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:49:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:49:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:49:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:49:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7606
2021/12/11 16:49:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:49:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:49:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:49:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:49:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:49:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:49:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:49:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/11 16:49:47 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/11 16:49:47 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:49:47 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/11 16:49:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:49:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:49:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:49:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:50:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:50:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (220.33s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/11 16:50:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:50:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:50:10 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:50:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:50:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:50:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:50:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:50:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:50:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:50:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:50:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:50:29 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:50:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:50:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:50:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:50:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:50:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:50:50 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:50:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:50:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:50:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:50:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:51:10 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:51:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:51:26 Rebalance progress: 12.5
2021/12/11 16:51:31 Rebalance progress: 12.5
2021/12/11 16:51:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:51:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:51:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:51:42 Rebalance progress: 12.5
2021/12/11 16:51:47 Rebalance progress: 12.5
2021/12/11 16:51:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:51:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:51:57 Rebalance progress: 12.5
2021/12/11 16:52:02 Rebalance progress: 12.5
2021/12/11 16:52:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:52:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:52:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:52:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:52:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:52:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:52:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:52:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:52:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:52:37 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:52:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:52:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:52:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:52:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9181
2021/12/11 16:52:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:52:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:52:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:53:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7218
2021/12/11 16:53:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7218
2021/12/11 16:53:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.49s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/11 16:53:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:53:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:53:16 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:53:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/11 16:53:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:53:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:53:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:53:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:53:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:53:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:53:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:53:34 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:53:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:53:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/11 16:53:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:53:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:53:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:53:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:53:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:53:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:53:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:53:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:54:01 Rebalance progress: 12.5
2021/12/11 16:54:06 Rebalance progress: 12.5
2021/12/11 16:54:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:54:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:54:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:54:26 Rebalance progress: 12.5
2021/12/11 16:54:31 Rebalance progress: 12.5
2021/12/11 16:54:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:54:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/11 16:54:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/11 16:54:41 Rebalance progress: 12.5
2021/12/11 16:54:46 Rebalance progress: 12.5
2021/12/11 16:54:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:54:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/11 16:54:56 Rebalance progress: 12.5
2021/12/11 16:55:01 Rebalance progress: 12.5
2021/12/11 16:55:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:55:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:55:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:55:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:55:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:55:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:55:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:55:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:55:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:55:37 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:55:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:55:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:55:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:55:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8577
2021/12/11 16:55:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:55:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:55:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:56:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3318
2021/12/11 16:56:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3318
2021/12/11 16:56:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.52s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/11 16:56:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/11 16:56:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/11 16:56:16 Rebalance progress: 100
2021/12/11 16:56:16 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:56:46 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/11 16:56:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/11 16:56:52 Removing node(s): [] from the cluster
2021/12/11 16:56:52 Starting bucket ops to source bucket
2021/12/11 16:56:57 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/11 16:57:02 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:57:07 Waiting for app: bucket_op_on_update to get deployed
2021/12/11 16:57:07 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/11 16:57:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/11 16:57:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:57:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:57:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:57:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:57:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:57:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/11 16:57:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/11 16:57:37 eventingSetup.go:46: No eventing-con process running
2021/12/11 16:57:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/11 16:57:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:57:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:57:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/11 16:57:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/11 16:57:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/11 16:57:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/11 16:58:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1231
2021/12/11 16:58:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1231
2021/12/11 16:58:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.50s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12093.943s

Rebalance tests took 3 hours, 21 minutes and 36 seconds.

Testing Succeeded


Version: versions-11.12.2021-09.14.cfg
Build Log: make-11.12.2021-09.14.log
Server Log: logs-11.12.2021-09.14.tar.gz

Finished