Building

Started building at 2021/12/02 03:23:37
Calculating base
Updating mirror
Basing run on 7.1.0-1821 ebf1dc831
Updating tree for run 02.12.2021-03.23
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a00b35f, changes since last good build: none
kv_engine is at d82da78, changes since last good build: 
 d82da7870 [Magma] Don't log every time we fill the scan buffer via CacheLookup
 58958d43f MB-49767: Fix and enhance logging in EPBucket::flushVBucket_UNLOCKED
 5e4b5ea6a MB-45185: Rename WantsDropped to CompactionCallbacks
 4c52a5b68 MB-49779: Optionally end OSO snapshot with SeqnoAdvance
ns_server is at 9756174, changes since last good build: none
couchstore is at 05c4320, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 6061aba, changes since last good build: none
Switching eventing to unstable
eventing is at e4e62e2, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 158037 at refs/changes/37/158037/4
Total 7 (delta 5), reused 7 (delta 5)
[unstable 1eb7253e] MB-47385: Move eventing-producer and cbevent go version to 1.16.5
 Author: AnkitPrabhu 
 Date: Thu Jul 22 11:08:10 2021 +0530
 3 files changed, 23 insertions(+), 4 deletions(-)
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable cfb38536] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable cfea69dd] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable a3a8823f] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166486 at refs/changes/86/166486/3
Total 7 (delta 2), reused 2 (delta 2)
[unstable 3bf0216a] MB-49148: Grant SystemXattrWrite privilege to on_behalf_of_ user
 Author: AnkitPrabhu 
 Date: Sat Nov 27 15:22:26 2021 +0530
 2 files changed, 31 insertions(+), 13 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 27f29348] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/02 03:39:49
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/02 03:41:32 Started functional
2021/12/02 03:41:42 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/02 03:41:55 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/02 03:41:55 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 03:42:00 Removing node(s): [] from the cluster
2021/12/02 03:42:05 Rebalance progress: 2.821180555555555
2021/12/02 03:42:10 Rebalance progress: 7.396556712962963
2021/12/02 03:42:15 Rebalance progress: 10.63368055555556
2021/12/02 03:42:20 Rebalance progress: 15.20905671296296
2021/12/02 03:42:26 Rebalance progress: 18.51851851851852
2021/12/02 03:42:30 Rebalance progress: 23.0396412037037
2021/12/02 03:42:35 Rebalance progress: 27.59693287037037
2021/12/02 03:42:40 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/02 03:42:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:43:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:43:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:43:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:43:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:44:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:44:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:44:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:44:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:45:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:45:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:45:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:45:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/02 03:46:01 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:46:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:46:07 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 03:46:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 03:46:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 03:46:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/02 03:46:22 App: TestAdvancedGetOps got deployed
2021/12/02 03:46:22 Starting bucket ops to source bucket
2021/12/02 03:46:27 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 03:46:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 03:46:37 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 03:46:37 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 03:46:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:46:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:46:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:46:42 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 03:46:48 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 03:46:53 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 03:46:58 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/02 03:46:58 App: TestAdvancedGetOps got un-deployed
2021/12/02 03:46:58 No eventing-con process running
2021/12/02 03:46:58 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/02 03:46:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:47:10 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 03:47:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 03:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:47:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:47:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 03:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (280.36s)
=== RUN   TestAdvancedInsertOps
2021/12/02 03:47:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:47:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:47:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:47:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:47:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:47:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:47:47 App: TestAdvancedInsertOps got deployed
2021/12/02 03:47:47 Starting bucket ops to source bucket
2021/12/02 03:47:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/02 03:47:58 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 03:48:03 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 03:48:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/02 03:48:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:48:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:48:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:48:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:48:14 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:48:19 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:48:24 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:48:24 App: TestAdvancedInsertOps got un-deployed
2021/12/02 03:48:24 No eventing-con process running
2021/12/02 03:48:24 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/02 03:48:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:48:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 03:48:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:48:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:48:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 03:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:48:47 Testing insert operation on source bucket
2021/12/02 03:48:47 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:48:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:48:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:48:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:49:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:49:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/02 03:49:08 App: TestAdvancedInsertOps got deployed
2021/12/02 03:49:08 Starting bucket ops to source bucket
2021/12/02 03:49:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/02 03:49:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/02 03:49:19 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/02 03:49:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:49:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:49:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:49:24 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:49:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:49:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:49:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/02 03:49:39 App: TestAdvancedInsertOps got un-deployed
2021/12/02 03:49:39 No eventing-con process running
2021/12/02 03:49:39 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/02 03:49:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:49:51 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/02 03:49:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/02 03:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:49:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:50:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.60s)
=== RUN   TestAdvancedUpsertOps
2021/12/02 03:50:03 Starting bucket ops to source bucket
2021/12/02 03:50:04 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:50:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:50:10 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:50:15 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:50:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:50:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:50:25 App: TestAdvancedUpsertOps got deployed
2021/12/02 03:50:25 Starting bucket ops to source bucket
2021/12/02 03:50:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 03:50:36 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/02 03:50:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:50:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:50:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:50:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:50:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:50:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:50:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:50:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:50:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/02 03:50:56 No eventing-con process running
2021/12/02 03:50:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/02 03:50:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:51:08 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/02 03:51:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:51:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:51:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:51:14 Testing upsert operation on source bucket
2021/12/02 03:51:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:51:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:51:21 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:51:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:51:31 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:51:36 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/02 03:51:36 App: TestAdvancedUpsertOps got deployed
2021/12/02 03:51:36 Starting bucket ops to source bucket
2021/12/02 03:51:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:51:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:51:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:51:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:51:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:51:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:51:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:51:57 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/02 03:51:57 App: TestAdvancedUpsertOps got un-deployed
2021/12/02 03:51:57 No eventing-con process running
2021/12/02 03:51:57 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/02 03:51:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:52:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.28s)
=== RUN   TestAdvancedReplaceOps
2021/12/02 03:52:11 Starting bucket ops to source bucket
2021/12/02 03:52:12 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:52:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:52:18 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:52:23 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:52:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:52:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:52:33 App: TestAdvancedReplaceOps got deployed
2021/12/02 03:52:33 Starting bucket ops to source bucket
2021/12/02 03:52:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 03:52:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 03:52:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:52:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:52:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:52:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:52:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:52:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:52:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:53:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:53:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/02 03:53:04 No eventing-con process running
2021/12/02 03:53:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/02 03:53:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:53:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 03:53:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:53:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:53:23 Testing upsert operation on source bucket
2021/12/02 03:53:23 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:53:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:53:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:53:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:53:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:53:44 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/02 03:53:44 App: TestAdvancedReplaceOps got deployed
2021/12/02 03:53:44 Starting bucket ops to source bucket
2021/12/02 03:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:53:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:53:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:53:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:53:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:53:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:54:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:54:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/02 03:54:06 App: TestAdvancedReplaceOps got un-deployed
2021/12/02 03:54:06 No eventing-con process running
2021/12/02 03:54:06 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/02 03:54:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:54:18 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/02 03:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:54:19 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:54:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (133.22s)
=== RUN   TestAdvancedDeleteOps
2021/12/02 03:54:24 Starting bucket ops to source bucket
2021/12/02 03:54:25 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:54:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:54:31 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:54:36 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:54:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:54:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:54:46 App: TestAdvancedDeleteOps got deployed
2021/12/02 03:54:46 Starting bucket ops to source bucket
2021/12/02 03:54:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 03:54:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:54:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:54:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:54:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:54:56 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:55:01 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:55:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:55:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:55:11 App: TestAdvancedDeleteOps got un-deployed
2021/12/02 03:55:11 No eventing-con process running
2021/12/02 03:55:11 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/02 03:55:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:55:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 03:55:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 03:55:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:55:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:55:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:55:34 Testing delete operation on source bucket
2021/12/02 03:55:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:55:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/02 03:55:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:55:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:55:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:55:56 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/02 03:55:56 App: TestAdvancedDeleteOps got deployed
2021/12/02 03:55:56 Starting bucket ops to source bucket
2021/12/02 03:55:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:55:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:55:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:55:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:56:01 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:56:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:56:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:56:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/02 03:56:16 App: TestAdvancedDeleteOps got un-deployed
2021/12/02 03:56:16 No eventing-con process running
2021/12/02 03:56:16 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/02 03:56:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:56:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:56:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.77s)
=== RUN   TestEnoentAdvancedGet
2021/12/02 03:56:30 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:56:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/02 03:56:36 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:56:41 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:56:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:56:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:56:51 App: TestEnoentAdvancedGet got deployed
2021/12/02 03:56:51 Starting bucket ops to source bucket
2021/12/02 03:56:57 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 03:57:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 03:57:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 03:57:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:57:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:57:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:57:07 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:57:12 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:57:17 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:57:22 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:57:22 App: TestEnoentAdvancedGet got un-deployed
2021/12/02 03:57:22 No eventing-con process running
2021/12/02 03:57:22 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/02 03:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:57:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 03:57:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 03:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:57:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:57:50 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/02 03:57:55 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/02 03:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:57:55 Testing get enoent operation on source bucket
2021/12/02 03:57:55 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:57:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/02 03:58:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:58:06 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:58:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:58:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/02 03:58:16 App: TestEnoentAdvancedGet got deployed
2021/12/02 03:58:16 Starting bucket ops to source bucket
2021/12/02 03:58:22 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/02 03:58:22 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/02 03:58:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:58:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:58:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:58:27 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:58:32 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:58:37 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:58:42 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/02 03:58:42 App: TestEnoentAdvancedGet got un-deployed
2021/12/02 03:58:42 No eventing-con process running
2021/12/02 03:58:42 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/02 03:58:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:58:54 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 03:58:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 03:58:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 03:59:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 03:59:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.22s)
=== RUN   TestEnoentAdvancedReplace
2021/12/02 03:59:05 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 03:59:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 03:59:11 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 03:59:16 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 03:59:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 03:59:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 03:59:27 App: TestEnoentAdvancedReplace got deployed
2021/12/02 03:59:27 Starting bucket ops to source bucket
2021/12/02 03:59:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 03:59:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 03:59:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 03:59:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 03:59:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 03:59:37 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 03:59:42 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 03:59:47 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 03:59:52 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 03:59:52 App: TestEnoentAdvancedReplace got un-deployed
2021/12/02 03:59:52 No eventing-con process running
2021/12/02 03:59:52 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/02 03:59:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:00:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:00:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:00:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:00:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:00:20 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/02 04:00:25 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/02 04:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:00:25 Testing get enoent operation on source bucket
2021/12/02 04:00:25 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:00:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 04:00:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 04:00:37 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 04:00:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 04:00:47 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/02 04:00:47 App: TestEnoentAdvancedReplace got deployed
2021/12/02 04:00:47 Starting bucket ops to source bucket
2021/12/02 04:00:52 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/02 04:00:52 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/02 04:00:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:00:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:00:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:00:57 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 04:01:02 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 04:01:07 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 04:01:12 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/02 04:01:12 App: TestEnoentAdvancedReplace got un-deployed
2021/12/02 04:01:12 No eventing-con process running
2021/12/02 04:01:12 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/02 04:01:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:01:24 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 04:01:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:01:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:01:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.07s)
=== RUN   TestEnoentAdvancedDelete
2021/12/02 04:01:35 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:01:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 04:01:42 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:01:47 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:01:52 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:01:57 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:01:57 App: TestEnoentAdvancedDelete got deployed
2021/12/02 04:01:57 Starting bucket ops to source bucket
2021/12/02 04:02:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:02:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:02:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:02:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:02:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:02:07 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:02:12 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:02:17 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:02:22 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:02:22 App: TestEnoentAdvancedDelete got un-deployed
2021/12/02 04:02:22 No eventing-con process running
2021/12/02 04:02:22 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/02 04:02:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:02:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:02:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:02:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:02:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:02:50 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/02 04:02:55 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/02 04:02:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:02:55 Testing get enoent operation on source bucket
2021/12/02 04:02:56 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:02:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 04:03:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:03:07 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:03:12 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:03:17 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/02 04:03:17 App: TestEnoentAdvancedDelete got deployed
2021/12/02 04:03:17 Starting bucket ops to source bucket
2021/12/02 04:03:22 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/02 04:03:22 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/02 04:03:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:03:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:03:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:03:27 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:03:32 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:03:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:03:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/02 04:03:42 App: TestEnoentAdvancedDelete got un-deployed
2021/12/02 04:03:42 No eventing-con process running
2021/12/02 04:03:42 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/02 04:03:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:03:54 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 04:03:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:03:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:04:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.13s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/02 04:04:06 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:04:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/02 04:04:12 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:04:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:04:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:04:27 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:04:27 App: TestAdvancedInsertKeyExist got deployed
2021/12/02 04:04:27 Starting bucket ops to source bucket
2021/12/02 04:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:04:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:04:27 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:04:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:04:32 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:04:37 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:04:42 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:04:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:04:47 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/02 04:04:47 No eventing-con process running
2021/12/02 04:04:47 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/02 04:04:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:04:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:04:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:05:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:05:01 Pumping document in the destination bucket
2021/12/02 04:05:01 Starting bucket ops to source bucket
2021/12/02 04:05:01 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:05:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/02 04:05:08 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:05:13 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:05:18 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:05:23 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/02 04:05:23 App: TestAdvancedInsertKeyExist got deployed
2021/12/02 04:05:23 Starting bucket ops to source bucket
2021/12/02 04:05:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:05:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:05:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:05:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:05:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:05:38 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:05:43 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:05:48 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:05:53 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/02 04:05:53 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/02 04:05:53 No eventing-con process running
2021/12/02 04:05:53 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/02 04:05:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:06:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:06:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:06:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:06:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (125.86s)
=== RUN   TestExipryGet
2021/12/02 04:06:11 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:06:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/02 04:06:18 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:06:23 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:06:28 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:06:33 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:06:33 App: TestExipryGet got deployed
2021/12/02 04:06:33 Starting bucket ops to source bucket
2021/12/02 04:07:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:07:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:07:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:07:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:07:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:07:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:07:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:07:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:07:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:07:43 App: TestExipryGet got un-deployed
2021/12/02 04:07:43 No eventing-con process running
2021/12/02 04:07:43 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/02 04:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:07:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:07:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:07:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:07:57 Pumping document in the destination bucket for expiry get
2021/12/02 04:07:57 Starting bucket ops to source bucket
2021/12/02 04:07:58 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:07:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/02 04:08:04 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:08:09 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:08:14 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:08:19 Waiting for app: TestExipryGet to get deployed
2021/12/02 04:08:19 App: TestExipryGet got deployed
2021/12/02 04:08:19 Starting bucket ops to source bucket
2021/12/02 04:09:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:09:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:09:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:09:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:09:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:09:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:09:14 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:09:19 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:09:24 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:09:29 Waiting for app: TestExipryGet to get un-deployed
2021/12/02 04:09:29 App: TestExipryGet got un-deployed
2021/12/02 04:09:29 No eventing-con process running
2021/12/02 04:09:29 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/02 04:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:09:41 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/02 04:09:46 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/02 04:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:09:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (221.28s)
=== RUN   TestExipryInsert
2021/12/02 04:09:53 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:09:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/02 04:09:59 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:10:04 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:10:09 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:10:14 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:10:14 App: TestExipryInsert got deployed
2021/12/02 04:10:14 Starting bucket ops to source bucket
2021/12/02 04:10:59 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/02 04:11:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:11:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:11:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:11:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:11:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:11:09 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:11:14 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:11:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:11:24 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:11:24 App: TestExipryInsert got un-deployed
2021/12/02 04:11:24 No eventing-con process running
2021/12/02 04:11:24 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/02 04:11:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:11:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:11:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:11:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:11:38 Testing on the destination bucket for expiry insert
2021/12/02 04:11:38 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:11:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/02 04:11:45 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:11:50 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:11:55 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:12:00 Waiting for app: TestExipryInsert to get deployed
2021/12/02 04:12:00 App: TestExipryInsert got deployed
2021/12/02 04:12:00 Starting bucket ops to source bucket
2021/12/02 04:12:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:12:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:12:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:12:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:12:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:12:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:12:55 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:13:00 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:13:05 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:13:10 Waiting for app: TestExipryInsert to get un-deployed
2021/12/02 04:13:10 App: TestExipryInsert got un-deployed
2021/12/02 04:13:10 No eventing-con process running
2021/12/02 04:13:10 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/02 04:13:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:13:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:13:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:13:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:13:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:13:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.94s)
=== RUN   TestExipryReplace
2021/12/02 04:13:34 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:13:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 04:13:40 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:13:45 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:13:50 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:13:55 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:13:55 App: TestExipryReplace got deployed
2021/12/02 04:13:55 Starting bucket ops to source bucket
2021/12/02 04:14:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:14:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:14:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:14:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:14:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:14:45 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:14:50 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:14:55 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:15:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:15:00 App: TestExipryReplace got un-deployed
2021/12/02 04:15:00 No eventing-con process running
2021/12/02 04:15:00 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/02 04:15:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:15:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:15:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:15:14 Testing on the destination bucket for expiry insert
2021/12/02 04:15:14 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:15:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 04:15:21 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:15:26 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:15:31 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:15:36 Waiting for app: TestExipryReplace to get deployed
2021/12/02 04:15:36 App: TestExipryReplace got deployed
2021/12/02 04:15:36 Starting bucket ops to source bucket
2021/12/02 04:16:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:16:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:16:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:16:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:16:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:16:26 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:16:31 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:16:36 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:16:41 Waiting for app: TestExipryReplace to get un-deployed
2021/12/02 04:16:41 App: TestExipryReplace got un-deployed
2021/12/02 04:16:41 No eventing-con process running
2021/12/02 04:16:41 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/02 04:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:16:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:16:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:16:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:16:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:17:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.82s)
=== RUN   TestExipryUpsert
2021/12/02 04:17:04 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:17:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/02 04:17:11 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:17:16 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:17:21 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:17:26 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:17:26 App: TestExipryUpsert got deployed
2021/12/02 04:17:26 Starting bucket ops to source bucket
2021/12/02 04:18:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:18:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:18:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:18:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:18:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:18:16 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:18:21 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:18:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:18:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:18:31 App: TestExipryUpsert got un-deployed
2021/12/02 04:18:31 No eventing-con process running
2021/12/02 04:18:31 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/02 04:18:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:18:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:18:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:18:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:18:45 Testing on the destination bucket for expiry insert
2021/12/02 04:18:45 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:18:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/02 04:18:51 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:18:56 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:19:01 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:19:06 Waiting for app: TestExipryUpsert to get deployed
2021/12/02 04:19:06 App: TestExipryUpsert got deployed
2021/12/02 04:19:06 Starting bucket ops to source bucket
2021/12/02 04:19:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:19:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:19:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:19:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:19:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:19:57 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:20:02 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:20:07 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:20:12 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/02 04:20:12 App: TestExipryUpsert got un-deployed
2021/12/02 04:20:12 No eventing-con process running
2021/12/02 04:20:12 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/02 04:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:20:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:20:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:20:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:20:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.83s)
=== RUN   TestCasReplace
2021/12/02 04:20:35 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:20:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 04:20:42 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:20:47 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:20:52 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:20:57 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:20:57 App: TestCasReplace got deployed
2021/12/02 04:20:57 Starting bucket ops to source bucket
2021/12/02 04:20:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:20:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:20:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:20:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:21:02 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:21:07 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:21:12 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:21:17 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:21:17 App: TestCasReplace got un-deployed
2021/12/02 04:21:17 No eventing-con process running
2021/12/02 04:21:17 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/02 04:21:19 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:21:29 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/02 04:21:34 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/02 04:21:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:21:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:21:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:21:41 Testing on the destination bucket for upsert with cas
2021/12/02 04:21:41 Starting bucket ops to source bucket
2021/12/02 04:21:41 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:21:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/02 04:21:47 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:21:52 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:21:57 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:22:02 Waiting for app: TestCasReplace to get deployed
2021/12/02 04:22:02 App: TestCasReplace got deployed
2021/12/02 04:22:02 Starting bucket ops to source bucket
2021/12/02 04:22:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:22:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:22:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:22:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:22:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:22:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:22:17 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:22:22 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:22:27 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:22:32 Waiting for app: TestCasReplace to get un-deployed
2021/12/02 04:22:32 App: TestCasReplace got un-deployed
2021/12/02 04:22:32 No eventing-con process running
2021/12/02 04:22:32 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/02 04:22:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:22:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:22:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:22:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:22:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (140.49s)
=== RUN   TestCasDelete
2021/12/02 04:22:56 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:22:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 04:23:02 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:23:07 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:23:12 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:23:17 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:23:17 App: TestCasDelete got deployed
2021/12/02 04:23:17 Starting bucket ops to source bucket
2021/12/02 04:23:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:23:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:23:17 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:23:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:23:22 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:23:27 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:23:32 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:23:37 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:23:37 App: TestCasDelete got un-deployed
2021/12/02 04:23:37 No eventing-con process running
2021/12/02 04:23:37 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/02 04:23:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:23:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:23:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:23:52 Testing on the destination bucket for delete with cas
2021/12/02 04:23:52 Starting bucket ops to source bucket
2021/12/02 04:23:52 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:23:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 04:23:58 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:24:03 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:24:08 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:24:13 Waiting for app: TestCasDelete to get deployed
2021/12/02 04:24:13 App: TestCasDelete got deployed
2021/12/02 04:24:13 Starting bucket ops to source bucket
2021/12/02 04:24:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:24:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:24:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:24:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:24:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 04:24:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:24:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:24:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:24:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:24:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/02 04:24:43 App: TestCasDelete got un-deployed
2021/12/02 04:24:43 No eventing-con process running
2021/12/02 04:24:43 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/02 04:24:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:24:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:24:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:25:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (126.08s)
=== RUN   TestCountersIncrement
2021/12/02 04:25:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 04:25:08 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 04:25:08 Removing node(s): [] from the cluster
2021/12/02 04:25:13 Rebalance progress: 100
2021/12/02 04:25:14 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:25:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/02 04:25:20 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:25:25 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:25:30 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:25:35 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:25:35 App: TestCountersIncrement got deployed
2021/12/02 04:25:35 Starting bucket ops to source bucket
2021/12/02 04:25:41 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/02 04:25:46 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 04:25:46 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/02 04:25:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:25:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:25:51 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:25:56 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:26:01 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:26:06 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:26:06 App: TestCountersIncrement got un-deployed
2021/12/02 04:26:06 No eventing-con process running
2021/12/02 04:26:06 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/02 04:26:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:26:18 Waiting for dst bucket item count to get to: 0 curr count: 607
2021/12/02 04:26:23 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/02 04:26:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:26:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:26:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 04:26:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:26:35 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:26:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/02 04:26:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:26:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:26:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:26:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/02 04:26:56 App: TestCountersIncrement got deployed
2021/12/02 04:26:56 Starting bucket ops to source bucket
2021/12/02 04:27:02 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/02 04:27:02 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/02 04:27:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:27:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:27:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:27:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:27:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:27:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/02 04:27:22 App: TestCountersIncrement got un-deployed
2021/12/02 04:27:22 No eventing-con process running
2021/12/02 04:27:22 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/02 04:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:27:34 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/02 04:27:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 04:27:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:27:51 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 04:27:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 04:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:27:56 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/02 04:28:01 Rebalance progress: 100
--- PASS: TestCountersIncrement (179.05s)
=== RUN   TestCountersDecrement
2021/12/02 04:28:01 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 04:28:01 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 04:28:02 Removing node(s): [] from the cluster
2021/12/02 04:28:07 Rebalance progress: 100
2021/12/02 04:28:07 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:28:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/02 04:28:13 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:28:18 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:28:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:28:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:28:28 App: TestCountersDecrement got deployed
2021/12/02 04:28:28 Starting bucket ops to source bucket
2021/12/02 04:28:34 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/02 04:28:39 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 04:28:44 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/02 04:28:44 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/02 04:28:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:28:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:28:49 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:28:54 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:28:59 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:29:04 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:29:04 App: TestCountersDecrement got un-deployed
2021/12/02 04:29:04 No eventing-con process running
2021/12/02 04:29:04 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/02 04:29:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:29:16 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/02 04:29:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:29:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:29:28 Waiting for dst bucket item count to get to: 0 curr count: 660
2021/12/02 04:29:33 Waiting for dst bucket item count to get to: 0 curr count: 148
2021/12/02 04:29:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:29:33 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:29:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/02 04:29:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:29:44 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:29:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:29:54 Waiting for app: TestCountersDecrement to get deployed
2021/12/02 04:29:54 App: TestCountersDecrement got deployed
2021/12/02 04:29:54 Starting bucket ops to source bucket
2021/12/02 04:30:00 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/02 04:30:05 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/02 04:30:05 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/02 04:30:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:30:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:30:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:30:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:30:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:30:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/02 04:30:25 App: TestCountersDecrement got un-deployed
2021/12/02 04:30:26 No eventing-con process running
2021/12/02 04:30:26 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/02 04:30:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:30:37 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/02 04:30:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:30:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:30:49 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 04:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:30:49 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/02 04:30:54 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.31s)
=== RUN   TestCacheGet
2021/12/02 04:30:54 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:30:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/02 04:31:01 Waiting for app: TestCacheGet to get deployed
2021/12/02 04:31:06 Waiting for app: TestCacheGet to get deployed
2021/12/02 04:31:11 Waiting for app: TestCacheGet to get deployed
2021/12/02 04:31:16 Waiting for app: TestCacheGet to get deployed
2021/12/02 04:31:16 App: TestCacheGet got deployed
2021/12/02 04:31:16 Starting bucket ops to source bucket
2021/12/02 04:31:21 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/02 04:31:26 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/02 04:31:31 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/02 04:31:36 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/02 04:31:41 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/02 04:31:41 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/02 04:31:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:31:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:31:46 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 04:31:51 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 04:31:56 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 04:32:01 Waiting for app: TestCacheGet to get un-deployed
2021/12/02 04:32:01 App: TestCacheGet got un-deployed
2021/12/02 04:32:01 No eventing-con process running
2021/12/02 04:32:01 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/02 04:32:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:32:13 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/02 04:32:18 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/02 04:32:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:32:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:32:30 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/02 04:32:35 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/02 04:32:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.36s)
=== RUN   TestCacheRYOW
2021/12/02 04:32:35 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:32:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/02 04:32:41 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 04:32:46 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 04:32:51 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 04:32:56 Waiting for app: TestCacheRYOW to get deployed
2021/12/02 04:32:56 App: TestCacheRYOW got deployed
2021/12/02 04:32:56 Starting bucket ops to source bucket
2021/12/02 04:33:01 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/02 04:33:01 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/02 04:33:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:33:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:33:06 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 04:33:11 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 04:33:16 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 04:33:21 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/02 04:33:21 App: TestCacheRYOW got un-deployed
2021/12/02 04:33:21 No eventing-con process running
2021/12/02 04:33:21 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/02 04:33:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:33:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:33:38 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 04:33:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:33:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:33:50 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 04:33:55 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 04:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.24s)
=== RUN   TestCacheExpiry
2021/12/02 04:33:55 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:33:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/02 04:34:01 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 04:34:06 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 04:34:11 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 04:34:16 Waiting for app: TestCacheExpiry to get deployed
2021/12/02 04:34:16 App: TestCacheExpiry got deployed
2021/12/02 04:34:16 Starting bucket ops to source bucket
2021/12/02 04:34:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:34:26 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 04:34:31 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 04:34:36 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 04:34:41 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 04:34:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:34:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:34:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:34:47 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 04:34:52 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 04:34:57 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 04:35:02 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/02 04:35:02 App: TestCacheExpiry got un-deployed
2021/12/02 04:35:02 No eventing-con process running
2021/12/02 04:35:02 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/02 04:35:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:35:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:35:18 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 04:35:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:35:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:35:30 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/02 04:35:35 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/02 04:35:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.40s)
=== RUN   TestCacheExpiryCustom
2021/12/02 04:35:35 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:35:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/02 04:35:42 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 04:35:47 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 04:35:52 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 04:35:57 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/02 04:35:57 App: TestCacheExpiryCustom got deployed
2021/12/02 04:35:57 Starting bucket ops to source bucket
2021/12/02 04:36:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:36:07 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 04:36:12 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/02 04:36:17 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 04:36:22 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/02 04:36:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:36:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:36:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:36:27 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 04:36:32 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 04:36:37 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 04:36:42 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/02 04:36:42 App: TestCacheExpiryCustom got un-deployed
2021/12/02 04:36:42 No eventing-con process running
2021/12/02 04:36:42 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/02 04:36:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:36:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:36:59 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/02 04:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:37:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:37:11 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/02 04:37:16 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/02 04:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.43s)
=== RUN   TestCacheOverflowCustom
2021/12/02 04:37:16 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:37:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/02 04:37:22 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 04:37:27 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 04:37:32 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 04:37:37 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/02 04:37:37 App: TestCacheOverflowCustom got deployed
2021/12/02 04:37:37 Starting bucket ops to source bucket
2021/12/02 04:37:42 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/02 04:37:47 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/02 04:37:52 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/02 04:37:57 Waiting for dst bucket item count to get to: 35 curr count: 26
2021/12/02 04:38:02 Waiting for dst bucket item count to get to: 35 curr count: 26
2021/12/02 04:38:07 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/02 04:38:12 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/02 04:38:12 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/02 04:38:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:38:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:38:17 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 04:38:22 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 04:38:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 04:38:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/02 04:38:32 App: TestCacheOverflowCustom got un-deployed
2021/12/02 04:38:32 No eventing-con process running
2021/12/02 04:38:32 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/02 04:38:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:38:44 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/02 04:38:49 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/02 04:38:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:38:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:39:01 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/02 04:39:06 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/02 04:39:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (110.25s)
=== RUN   TestStrictMode
2021/12/02 04:39:06 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:39:07 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/02 04:39:07 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:39:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/02 04:39:13 Waiting for app: TestDataTypes to get deployed
2021/12/02 04:39:18 Waiting for app: TestDataTypes to get deployed
2021/12/02 04:39:23 Waiting for app: TestDataTypes to get deployed
2021/12/02 04:39:28 Waiting for app: TestDataTypes to get deployed
2021/12/02 04:39:28 App: TestDataTypes got deployed
2021/12/02 04:39:28 Starting bucket ops to source bucket
2021/12/02 04:39:33 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/02 04:39:38 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/02 04:39:43 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/02 04:39:43 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/02 04:39:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:39:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:39:48 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 04:39:53 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 04:39:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 04:40:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/02 04:40:03 App: TestDataTypes got un-deployed
2021/12/02 04:40:03 No eventing-con process running
2021/12/02 04:40:04 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/02 04:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:40:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 04:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:40:27 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/02 04:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.13s)
=== RUN   TestEnoentGet
2021/12/02 04:40:27 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:40:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/02 04:40:33 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:40:38 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:40:43 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:40:48 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:40:48 App: TestEnoentGet got deployed
2021/12/02 04:40:48 Starting bucket ops to source bucket
2021/12/02 04:40:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:40:58 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 04:40:58 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:40:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:40:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:41:04 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:41:09 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:41:14 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:41:19 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:41:19 App: TestEnoentGet got un-deployed
2021/12/02 04:41:19 No eventing-con process running
2021/12/02 04:41:19 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/02 04:41:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:41:30 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/02 04:41:35 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/02 04:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:41:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:41:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:41:47 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:41:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/02 04:41:54 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:41:59 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:42:04 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:42:09 Waiting for app: TestEnoentGet to get deployed
2021/12/02 04:42:09 App: TestEnoentGet got deployed
2021/12/02 04:42:09 Starting bucket ops to source bucket
2021/12/02 04:42:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:42:19 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 04:42:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:42:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:42:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:42:24 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:42:29 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:42:34 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:42:39 Waiting for app: TestEnoentGet to get un-deployed
2021/12/02 04:42:39 App: TestEnoentGet got un-deployed
2021/12/02 04:42:39 No eventing-con process running
2021/12/02 04:42:39 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/02 04:42:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:42:51 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/02 04:42:56 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/02 04:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:42:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:43:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:43:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.42s)
=== RUN   TestEnoentDelete
2021/12/02 04:43:08 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:43:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 04:43:14 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:43:19 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:43:24 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:43:29 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:43:29 App: TestEnoentDelete got deployed
2021/12/02 04:43:29 Starting bucket ops to source bucket
2021/12/02 04:43:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:43:39 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 04:43:39 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:43:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:43:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:43:44 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:43:49 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:43:54 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:43:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:43:59 App: TestEnoentDelete got un-deployed
2021/12/02 04:43:59 No eventing-con process running
2021/12/02 04:43:59 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/02 04:44:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:44:11 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/02 04:44:16 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/02 04:44:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:44:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:44:28 Waiting for dst bucket item count to get to: 0 curr count: 55
2021/12/02 04:44:33 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/02 04:44:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:44:33 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:44:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 04:44:39 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:44:44 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:44:49 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:44:54 Waiting for app: TestEnoentDelete to get deployed
2021/12/02 04:44:54 App: TestEnoentDelete got deployed
2021/12/02 04:44:54 Starting bucket ops to source bucket
2021/12/02 04:44:59 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:45:04 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 04:45:04 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:45:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:45:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:45:09 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:45:14 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:45:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:45:24 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/02 04:45:24 App: TestEnoentDelete got un-deployed
2021/12/02 04:45:24 No eventing-con process running
2021/12/02 04:45:24 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/02 04:45:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:45:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:45:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:45:48 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/02 04:45:53 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/02 04:45:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (165.36s)
=== RUN   TestError
2021/12/02 04:46:03 Waiting for app: TestError to get un-deployed
2021/12/02 04:46:03 App: TestError got un-deployed
2021/12/02 04:46:03 No eventing-con process running
2021/12/02 04:46:03 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/02 04:46:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:46:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:46:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:46:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:46:17 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:46:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/02 04:46:23 Waiting for app: TestError to get deployed
2021/12/02 04:46:28 Waiting for app: TestError to get deployed
2021/12/02 04:46:33 Waiting for app: TestError to get deployed
2021/12/02 04:46:38 Waiting for app: TestError to get deployed
2021/12/02 04:46:38 App: TestError got deployed
2021/12/02 04:46:38 Starting bucket ops to source bucket
2021/12/02 04:46:43 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/02 04:46:48 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/02 04:46:53 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/02 04:46:53 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/02 04:46:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:46:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:46:58 Waiting for app: TestError to get un-deployed
2021/12/02 04:47:03 Waiting for app: TestError to get un-deployed
2021/12/02 04:47:08 Waiting for app: TestError to get un-deployed
2021/12/02 04:47:13 Waiting for app: TestError to get un-deployed
2021/12/02 04:47:13 App: TestError got un-deployed
2021/12/02 04:47:13 No eventing-con process running
2021/12/02 04:47:13 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/02 04:47:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:47:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:47:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:47:37 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/02 04:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.97s)
=== RUN   TestCRLF
2021/12/02 04:47:47 Waiting for app: TestCRLF to get un-deployed
2021/12/02 04:47:47 App: TestCRLF got un-deployed
2021/12/02 04:47:47 No eventing-con process running
2021/12/02 04:47:47 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/02 04:47:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:48:01 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:48:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/02 04:48:07 Waiting for app: TestCRLF to get deployed
2021/12/02 04:48:12 Waiting for app: TestCRLF to get deployed
2021/12/02 04:48:17 Waiting for app: TestCRLF to get deployed
2021/12/02 04:48:22 Waiting for app: TestCRLF to get deployed
2021/12/02 04:48:22 App: TestCRLF got deployed
2021/12/02 04:48:22 Starting bucket ops to source bucket
2021/12/02 04:48:27 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:48:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 04:48:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 04:48:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:48:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:48:37 Waiting for app: TestCRLF to get un-deployed
2021/12/02 04:48:42 Waiting for app: TestCRLF to get un-deployed
2021/12/02 04:48:47 Waiting for app: TestCRLF to get un-deployed
2021/12/02 04:48:52 Waiting for app: TestCRLF to get un-deployed
2021/12/02 04:48:52 App: TestCRLF got un-deployed
2021/12/02 04:48:52 No eventing-con process running
2021/12/02 04:48:52 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/02 04:48:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:49:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 04:49:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 04:49:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:49:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:49:21 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/02 04:49:26 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/02 04:49:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (109.01s)
=== RUN   TestImportExport
2021/12/02 04:49:36 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 04:49:36 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/02 04:49:36 No eventing-con process running
2021/12/02 04:49:36 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/02 04:49:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:49:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:49:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:49:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:49:55 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 04:49:55 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/02 04:49:55 No eventing-con process running
2021/12/02 04:49:55 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/02 04:49:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:50:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:50:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:50:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:50:08 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:50:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 04:50:10 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:50:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 04:50:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 04:50:21 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 04:50:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 04:50:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/02 04:50:31 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/02 04:50:36 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/02 04:50:41 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/02 04:50:46 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/02 04:50:46 App: TestImportExport_bucket_op_on_update got deployed
2021/12/02 04:50:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:50:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 04:50:56 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 04:51:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 04:51:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 04:51:06 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/02 04:51:06 No eventing-con process running
2021/12/02 04:51:06 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/02 04:51:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:51:11 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 04:51:16 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 04:51:21 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 04:51:26 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 04:51:26 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/02 04:51:26 No eventing-con process running
2021/12/02 04:51:26 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/02 04:51:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/02 04:51:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/02 04:51:43 No eventing-con process running
2021/12/02 04:51:44 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/02 04:51:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:51:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:51:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:51:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:52:02 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/02 04:52:02 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/02 04:52:02 No eventing-con process running
2021/12/02 04:52:02 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/02 04:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:52:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.92s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/02 04:52:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:52:26 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 04:52:26 No eventing-con process running
2021/12/02 04:52:26 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/02 04:52:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:52:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:52:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:52:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:52:39 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:52:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 04:52:41 Starting bucket ops to source bucket
2021/12/02 04:52:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:52:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:52:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:53:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:53:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 04:53:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 04:53:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 04:53:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:53:09 Undeploying app: bucket_op_on_update
2021/12/02 04:53:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:53:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:53:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:53:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:53:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:53:29 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 04:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:53:29 No eventing-con process running
2021/12/02 04:53:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:53:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:54:13 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:54:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 04:54:14 Starting bucket ops to source bucket
2021/12/02 04:54:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:54:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:54:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:54:37 Waiting for dst bucket item count to get to: 5000 curr count: 980
2021/12/02 04:54:42 Waiting for dst bucket item count to get to: 5000 curr count: 3480
2021/12/02 04:54:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 04:54:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 04:54:47 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 04:54:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:54:47 Undeploying app: bucket_op_on_update
2021/12/02 04:54:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:54:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:54:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:55:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:55:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:55:08 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 04:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:55:08 No eventing-con process running
2021/12/02 04:55:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:55:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:55:52 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:55:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 04:55:53 Starting bucket ops to source bucket
2021/12/02 04:56:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:56:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:56:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:56:16 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 04:56:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 04:56:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 04:56:21 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 04:56:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:56:21 Undeploying app: bucket_op_on_update
2021/12/02 04:56:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:56:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:56:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:56:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:56:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:56:41 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 04:56:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:56:41 No eventing-con process running
2021/12/02 04:56:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:56:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:57:25 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:57:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 04:57:26 Starting bucket ops to source bucket
2021/12/02 04:57:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:57:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:57:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:57:49 Waiting for dst bucket item count to get to: 5000 curr count: 1799
2021/12/02 04:57:54 Waiting for dst bucket item count to get to: 5000 curr count: 1799
2021/12/02 04:57:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 04:57:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 04:57:59 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 04:57:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:57:59 Undeploying app: bucket_op_on_update
2021/12/02 04:57:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:58:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:58:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:58:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:58:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:58:19 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 04:58:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:58:19 No eventing-con process running
2021/12/02 04:58:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:58:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 04:59:04 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 04:59:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 04:59:05 Starting bucket ops to source bucket
2021/12/02 04:59:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:59:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:59:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 04:59:28 Waiting for dst bucket item count to get to: 5000 curr count: 206
2021/12/02 04:59:33 Waiting for dst bucket item count to get to: 5000 curr count: 2706
2021/12/02 04:59:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 04:59:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/02 04:59:38 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/02 04:59:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 04:59:38 Undeploying app: bucket_op_on_update
2021/12/02 04:59:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 04:59:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:59:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:59:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:59:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/02 04:59:58 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/02 04:59:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 04:59:58 No eventing-con process running
2021/12/02 05:00:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:00:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:00:42 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (506.35s)
=== RUN   TestOnUpdateN1QLOp
2021/12/02 05:00:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 05:00:52 App: TestOnUpdateN1QLOp got un-deployed
2021/12/02 05:00:52 No eventing-con process running
2021/12/02 05:00:52 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/02 05:00:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:00:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:01:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:01:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:01:06 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:01:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/02 05:01:07 Starting bucket ops to source bucket
2021/12/02 05:01:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:01:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:01:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:01:30 Waiting for dst bucket item count to get to: 5000 curr count: 213
2021/12/02 05:01:35 Waiting for dst bucket item count to get to: 5000 curr count: 213
2021/12/02 05:01:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:01:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:01:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:01:40 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 05:01:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 05:01:50 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 05:01:55 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/02 05:01:55 App: TestOnUpdateN1QLOp got un-deployed
2021/12/02 05:01:55 No eventing-con process running
2021/12/02 05:01:55 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/02 05:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:02:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:02:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:02:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:02:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:02:24 Waiting for dst bucket item count to get to: 0 curr count: 2489
2021/12/02 05:02:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.83s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/02 05:02:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 05:02:34 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/02 05:02:34 No eventing-con process running
2021/12/02 05:02:34 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/02 05:02:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:02:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:02:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:02:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:02:48 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:02:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 05:02:49 Starting bucket ops to source bucket
2021/12/02 05:02:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:03:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:03:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:03:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:03:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 05:03:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:03:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:03:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:03:22 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 05:03:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 05:03:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 05:03:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/02 05:03:37 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/02 05:03:37 No eventing-con process running
2021/12/02 05:03:37 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/02 05:03:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:03:49 Waiting for dst bucket item count to get to: 0 curr count: 4700
2021/12/02 05:03:54 Waiting for dst bucket item count to get to: 0 curr count: 2200
2021/12/02 05:03:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:03:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:04:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.75s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/02 05:04:16 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 05:04:16 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 05:04:16 No eventing-con process running
2021/12/02 05:04:16 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/02 05:04:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:04:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:04:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:04:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:04:29 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:04:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 05:04:31 Starting bucket ops to source bucket
2021/12/02 05:04:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:04:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:04:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:04:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:04:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:04:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:04:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:04:59 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 05:05:04 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 05:05:09 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 05:05:14 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 05:05:14 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 05:05:14 No eventing-con process running
2021/12/02 05:05:14 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/02 05:05:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:05:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:05:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:05:38 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (91.84s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/02 05:05:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 05:05:48 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/02 05:05:48 No eventing-con process running
2021/12/02 05:05:48 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/02 05:05:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:05:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:05:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:06:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:06:01 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:06:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/02 05:06:02 Starting bucket ops to source bucket
2021/12/02 05:06:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:06:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:06:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:06:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:06:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:06:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:06:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:06:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 05:06:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 05:06:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 05:06:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/02 05:06:46 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/02 05:06:46 No eventing-con process running
2021/12/02 05:06:46 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/02 05:06:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:06:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:06:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:06:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:07:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:07:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.84s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/02 05:07:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 05:07:24 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/02 05:07:24 No eventing-con process running
2021/12/02 05:07:24 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/02 05:07:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:07:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:07:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:07:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:07:38 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:07:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/02 05:07:40 Starting bucket ops to source bucket
2021/12/02 05:07:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:07:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:07:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:08:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:08:08 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 05:08:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:08:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:08:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:08:13 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 05:08:18 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 05:08:23 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 05:08:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/02 05:08:28 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/02 05:08:28 No eventing-con process running
2021/12/02 05:08:28 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/02 05:08:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:08:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:08:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:08:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:08:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.23s)
=== RUN   TestOnDeleteBucketOp
2021/12/02 05:09:07 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 05:09:07 App: TestOnDeleteBucketOp got un-deployed
2021/12/02 05:09:07 No eventing-con process running
2021/12/02 05:09:07 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/02 05:09:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:09:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:09:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:09:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:09:20 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:09:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 05:09:21 Starting bucket ops to source bucket
2021/12/02 05:09:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:09:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:09:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:09:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:09:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:09:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:09:47 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 05:09:52 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 05:09:57 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 05:10:02 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/02 05:10:02 App: TestOnDeleteBucketOp got un-deployed
2021/12/02 05:10:02 No eventing-con process running
2021/12/02 05:10:02 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/02 05:10:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:10:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:10:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:10:22 Waiting for dst bucket item count to get to: 0 curr count: 4972
2021/12/02 05:10:27 Waiting for dst bucket item count to get to: 0 curr count: 4972
2021/12/02 05:10:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.00s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/02 05:10:37 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 05:10:37 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/02 05:10:37 No eventing-con process running
2021/12/02 05:10:37 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/02 05:10:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:10:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:10:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:10:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:10:50 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:10:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/02 05:10:51 Starting bucket ops to source bucket
2021/12/02 05:11:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:11:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:11:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:11:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:11:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:11:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:11:18 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 05:11:23 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 05:11:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 05:11:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/02 05:11:33 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/02 05:11:33 No eventing-con process running
2021/12/02 05:11:33 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/02 05:11:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:11:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:11:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:11:52 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:11:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.18s)
=== RUN   TestTimerBucketOp
2021/12/02 05:12:07 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 05:12:07 App: TestTimerBucketOp got un-deployed
2021/12/02 05:12:07 No eventing-con process running
2021/12/02 05:12:07 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/02 05:12:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:12:15 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:12:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:12:20 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:12:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 05:12:22 Starting bucket ops to source bucket
2021/12/02 05:12:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:12:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:12:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:12:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:12:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:12:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:13:00 Waiting for dst bucket item count to get to: 5000 curr count: 1387
2021/12/02 05:13:05 Waiting for dst bucket item count to get to: 5000 curr count: 1387
2021/12/02 05:13:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:13:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:13:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:13:10 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 05:13:15 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 05:13:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 05:13:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/02 05:13:25 App: TestTimerBucketOp got un-deployed
2021/12/02 05:13:25 No eventing-con process running
2021/12/02 05:13:25 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/02 05:13:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:13:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:13:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:13:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:13:49 Waiting for dst bucket item count to get to: 0 curr count: 4424
2021/12/02 05:13:54 Waiting for dst bucket item count to get to: 0 curr count: 1924
2021/12/02 05:13:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.83s)
=== RUN   TestTimerInPastBucketOp
2021/12/02 05:14:04 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 05:14:04 App: TestTimerInPastBucketOp got un-deployed
2021/12/02 05:14:04 No eventing-con process running
2021/12/02 05:14:04 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/02 05:14:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:14:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:14:17 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:14:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 05:14:18 Starting bucket ops to source bucket
2021/12/02 05:14:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:14:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:14:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:14:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:14:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:14:52 Waiting for dst bucket item count to get to: 5000 curr count: 6
2021/12/02 05:14:57 Waiting for dst bucket item count to get to: 5000 curr count: 2455
2021/12/02 05:15:02 Waiting for dst bucket item count to get to: 5000 curr count: 4949
2021/12/02 05:15:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:15:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:15:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:15:07 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 05:15:12 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 05:15:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 05:15:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/02 05:15:22 App: TestTimerInPastBucketOp got un-deployed
2021/12/02 05:15:22 No eventing-con process running
2021/12/02 05:15:22 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/02 05:15:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:15:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:15:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 05:15:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:15:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:15:50 Waiting for dst bucket item count to get to: 0 curr count: 4025
2021/12/02 05:15:55 Waiting for dst bucket item count to get to: 0 curr count: 4025
2021/12/02 05:15:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.85s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/02 05:16:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:16:06 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 05:16:06 No eventing-con process running
2021/12/02 05:16:06 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/02 05:16:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:16:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:16:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:16:19 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:16:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 05:16:20 Starting bucket ops to source bucket
2021/12/02 05:16:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:16:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:16:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:16:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:16:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:16:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:16:58 Waiting for dst bucket item count to get to: 5000 curr count: 2896
2021/12/02 05:17:03 Waiting for dst bucket item count to get to: 5000 curr count: 2896
2021/12/02 05:17:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:17:08 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 05:17:08 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 05:17:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:17:08 Undeploying app: bucket_op_with_timer
2021/12/02 05:17:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:17:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:17:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:17:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:17:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:17:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 05:17:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:17:29 No eventing-con process running
2021/12/02 05:17:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:17:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:18:13 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:18:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 05:18:14 Starting bucket ops to source bucket
2021/12/02 05:18:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:18:57 Waiting for dst bucket item count to get to: 5000 curr count: 1817
2021/12/02 05:19:02 Waiting for dst bucket item count to get to: 5000 curr count: 4317
2021/12/02 05:19:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:19:07 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 05:19:07 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 05:19:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:19:07 Undeploying app: bucket_op_with_timer
2021/12/02 05:19:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:19:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:19:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:19:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:19:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:19:27 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 05:19:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:19:27 No eventing-con process running
2021/12/02 05:19:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:19:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:20:11 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:20:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 05:20:12 Starting bucket ops to source bucket
2021/12/02 05:20:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:20:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:20:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:20:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:20:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:20:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:20:50 Waiting for dst bucket item count to get to: 5000 curr count: 597
2021/12/02 05:20:55 Waiting for dst bucket item count to get to: 5000 curr count: 597
2021/12/02 05:20:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:21:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 05:21:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 05:21:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:21:00 Undeploying app: bucket_op_with_timer
2021/12/02 05:21:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:21:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:21:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:21:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:21:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:21:21 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 05:21:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:21:21 No eventing-con process running
2021/12/02 05:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:22:05 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:22:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 05:22:06 Starting bucket ops to source bucket
2021/12/02 05:22:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:22:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:22:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:22:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:22:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:22:39 Waiting for dst bucket item count to get to: 5000 curr count: 454
2021/12/02 05:22:44 Waiting for dst bucket item count to get to: 5000 curr count: 454
2021/12/02 05:22:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:22:49 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 05:22:49 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 05:22:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:22:49 Undeploying app: bucket_op_with_timer
2021/12/02 05:22:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:22:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:22:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:23:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:23:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:23:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 05:23:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:23:09 No eventing-con process running
2021/12/02 05:23:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:23:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:23:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:23:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 05:23:54 Starting bucket ops to source bucket
2021/12/02 05:24:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:24:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/02 05:24:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:24:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/02 05:24:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/02 05:24:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:24:42 Undeploying app: bucket_op_with_timer
2021/12/02 05:24:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:24:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:24:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:24:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:25:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/02 05:25:03 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/02 05:25:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:25:03 No eventing-con process running
2021/12/02 05:25:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:25:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:25:47 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (591.26s)
=== RUN   TestMultipleHandlers
2021/12/02 05:25:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 05:25:57 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/02 05:25:57 No eventing-con process running
2021/12/02 05:25:57 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/02 05:25:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:26:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:26:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:26:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:26:15 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 05:26:15 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/02 05:26:15 No eventing-con process running
2021/12/02 05:26:15 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/02 05:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:26:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:26:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:26:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:26:29 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:26:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 05:26:30 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:26:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/02 05:26:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 05:26:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 05:26:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 05:26:51 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/02 05:26:51 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/02 05:26:56 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/02 05:27:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/02 05:27:06 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/02 05:27:06 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/02 05:27:06 Starting bucket ops to source bucket
2021/12/02 05:27:16 Waiting for dst bucket item count to get to: 5000 curr count: 1858
2021/12/02 05:27:21 Waiting for dst bucket item count to get to: 5000 curr count: 1858
2021/12/02 05:27:21 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:27:21 Starting bucket ops to source bucket
2021/12/02 05:27:36 Waiting for dst bucket item count to get to: 10000 curr count: 6785
2021/12/02 05:27:41 Waiting for dst bucket item count to get to: 10000 curr count: 6785
2021/12/02 05:27:41 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 05:27:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:27:41 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:27:42 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:27:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:27:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 05:27:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 05:27:57 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 05:28:02 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/02 05:28:02 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/02 05:28:02 No eventing-con process running
2021/12/02 05:28:02 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/02 05:28:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:28:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:28:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:28:21 Waiting for dst bucket item count to get to: 0 curr count: 8414
2021/12/02 05:28:26 Waiting for dst bucket item count to get to: 0 curr count: 8414
2021/12/02 05:28:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:28:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:28:31 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 05:28:36 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 05:28:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 05:28:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/02 05:28:46 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/02 05:28:46 No eventing-con process running
2021/12/02 05:28:46 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/02 05:28:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:28:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:28:55 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:29:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (193.06s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/02 05:29:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 05:29:10 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/02 05:29:10 No eventing-con process running
2021/12/02 05:29:10 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/02 05:29:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:29:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:29:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:29:23 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:29:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 05:29:25 Starting bucket ops to source bucket
2021/12/02 05:29:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:29:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:29:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:29:48 Waiting for dst bucket item count to get to: 5000 curr count: 633
2021/12/02 05:29:53 Waiting for dst bucket item count to get to: 5000 curr count: 3133
2021/12/02 05:29:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:29:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:29:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 05:29:58 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 05:29:58 Pausing app: bucket_op_on_update
2021/12/02 05:29:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:30:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 05:30:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 05:30:03 Resuming app: bucket_op_on_update
2021/12/02 05:30:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:30:04 Starting bucket ops to source bucket
2021/12/02 05:30:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:30:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:30:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:30:27 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/02 05:30:27 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 05:30:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:30:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 05:30:32 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 05:30:32 Pausing app: bucket_op_on_update
2021/12/02 05:30:32 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:30:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 05:30:37 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 05:30:37 Resuming app: bucket_op_on_update
2021/12/02 05:30:38 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:30:38 Starting bucket ops to source bucket
2021/12/02 05:30:46 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 05:30:51 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 05:30:51 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/02 05:30:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:30:56 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 05:30:56 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 05:30:56 Pausing app: bucket_op_on_update
2021/12/02 05:30:56 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:31:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 05:31:01 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 05:31:01 Resuming app: bucket_op_on_update
2021/12/02 05:31:03 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:31:03 Starting bucket ops to source bucket
2021/12/02 05:31:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:31:16 Waiting for dst bucket item count to get to: 20000 curr count: 15002
2021/12/02 05:31:21 Waiting for dst bucket item count to get to: 20000 curr count: 15002
2021/12/02 05:31:26 Waiting for dst bucket item count to get to: 20000 curr count: 17502
2021/12/02 05:31:26 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/02 05:31:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:31:31 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 05:31:31 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 05:31:31 Pausing app: bucket_op_on_update
2021/12/02 05:31:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:31:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 05:31:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 05:31:36 Resuming app: bucket_op_on_update
2021/12/02 05:31:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:31:37 Starting bucket ops to source bucket
2021/12/02 05:31:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:31:50 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:31:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:31:55 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/02 05:31:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:32:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/02 05:32:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/02 05:32:00 Pausing app: bucket_op_on_update
2021/12/02 05:32:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:32:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/02 05:32:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/02 05:32:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:32:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 05:32:15 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 05:32:20 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 05:32:25 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/02 05:32:25 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/02 05:32:25 No eventing-con process running
2021/12/02 05:32:25 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/02 05:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:32:37 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 05:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:32:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:32:49 Waiting for dst bucket item count to get to: 0 curr count: 24761
2021/12/02 05:32:54 Waiting for dst bucket item count to get to: 0 curr count: 12261
2021/12/02 05:32:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.08s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/02 05:33:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 05:33:04 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/02 05:33:04 No eventing-con process running
2021/12/02 05:33:04 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/02 05:33:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:33:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:33:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:33:18 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:33:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 05:33:19 Starting bucket ops to source bucket
2021/12/02 05:33:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:33:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:33:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:33:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:33:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:33:52 Waiting for dst bucket item count to get to: 5000 curr count: 23
2021/12/02 05:33:57 Waiting for dst bucket item count to get to: 5000 curr count: 2523
2021/12/02 05:33:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:33:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:34:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 05:34:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 05:34:02 Pausing app: bucket_op_with_timer
2021/12/02 05:34:02 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:34:07 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 05:34:07 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 05:34:07 Resuming app: bucket_op_with_timer
2021/12/02 05:34:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:34:08 Starting bucket ops to source bucket
2021/12/02 05:34:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:34:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:34:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:34:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:34:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:34:41 Waiting for dst bucket item count to get to: 10000 curr count: 5053
2021/12/02 05:34:46 Waiting for dst bucket item count to get to: 10000 curr count: 7553
2021/12/02 05:34:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 05:34:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:34:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 05:34:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 05:34:51 Pausing app: bucket_op_with_timer
2021/12/02 05:34:51 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:34:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 05:34:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 05:34:56 Resuming app: bucket_op_with_timer
2021/12/02 05:34:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:34:58 Starting bucket ops to source bucket
2021/12/02 05:35:06 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 05:35:11 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 05:35:16 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 05:35:21 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/02 05:35:26 Waiting for dst bucket item count to get to: 15000 curr count: 10625
2021/12/02 05:35:31 Waiting for dst bucket item count to get to: 15000 curr count: 10625
2021/12/02 05:35:31 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/02 05:35:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:35:36 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 05:35:36 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 05:35:36 Pausing app: bucket_op_with_timer
2021/12/02 05:35:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:35:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 05:35:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 05:35:41 Resuming app: bucket_op_with_timer
2021/12/02 05:35:42 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:35:42 Starting bucket ops to source bucket
2021/12/02 05:35:50 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:35:55 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:36:00 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:36:05 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:36:10 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:36:15 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/02 05:36:20 Waiting for dst bucket item count to get to: 20000 curr count: 18836
2021/12/02 05:36:25 Waiting for dst bucket item count to get to: 20000 curr count: 18836
2021/12/02 05:36:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/02 05:36:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:36:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 05:36:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 05:36:30 Pausing app: bucket_op_with_timer
2021/12/02 05:36:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:36:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 05:36:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 05:36:35 Resuming app: bucket_op_with_timer
2021/12/02 05:36:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:36:36 Starting bucket ops to source bucket
2021/12/02 05:36:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:36:49 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:36:54 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:36:59 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:37:04 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/02 05:37:09 Waiting for dst bucket item count to get to: 25000 curr count: 22489
2021/12/02 05:37:14 Waiting for dst bucket item count to get to: 25000 curr count: 22489
2021/12/02 05:37:14 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/02 05:37:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:37:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/02 05:37:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/02 05:37:19 Pausing app: bucket_op_with_timer
2021/12/02 05:37:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:37:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/02 05:37:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/02 05:37:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:37:30 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 05:37:35 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 05:37:40 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 05:37:45 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/02 05:37:45 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/02 05:37:45 No eventing-con process running
2021/12/02 05:37:45 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/02 05:37:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:37:56 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 05:37:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:37:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:38:08 Waiting for dst bucket item count to get to: 0 curr count: 19270
2021/12/02 05:38:13 Waiting for dst bucket item count to get to: 0 curr count: 6770
2021/12/02 05:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.27s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/02 05:38:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 05:38:23 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/02 05:38:23 No eventing-con process running
2021/12/02 05:38:23 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/02 05:38:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:38:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:38:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:38:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:38:37 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:38:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/02 05:38:43 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 05:38:48 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 05:38:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 05:38:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/02 05:38:58 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/02 05:38:58 Starting bucket ops to source bucket
2021/12/02 05:39:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:39:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:39:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:39:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:39:27 Waiting for dst bucket item count to get to: 5000 curr count: 823
2021/12/02 05:39:32 Waiting for dst bucket item count to get to: 5000 curr count: 2538
2021/12/02 05:39:37 Waiting for dst bucket item count to get to: 5000 curr count: 4215
2021/12/02 05:39:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:39:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:39:37 Pausing app: bucket_op_with_timer
2021/12/02 05:39:37 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:39:42 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/02 05:39:43 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/02 05:39:43 Resuming app: bucket_op_with_timer
2021/12/02 05:39:44 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:39:44 Starting bucket ops to source bucket
2021/12/02 05:39:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:40:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:40:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:40:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:40:16 Waiting for dst bucket item count to get to: 10000 curr count: 6348
2021/12/02 05:40:21 Waiting for dst bucket item count to get to: 10000 curr count: 8848
2021/12/02 05:40:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 05:40:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:40:26 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 05:40:31 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 05:40:36 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 05:40:41 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/02 05:40:41 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/02 05:40:41 No eventing-con process running
2021/12/02 05:40:41 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/02 05:40:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:40:53 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 05:40:58 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:40:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:41:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:41:10 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:41:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.61s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/02 05:41:25 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 05:41:25 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/02 05:41:25 No eventing-con process running
2021/12/02 05:41:25 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/02 05:41:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:41:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:41:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:41:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:41:38 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:41:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 05:41:45 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 05:41:50 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 05:41:55 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 05:42:00 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/02 05:42:00 App: TestPauseResumeWithEventingReb got deployed
2021/12/02 05:42:00 Starting bucket ops to source bucket
2021/12/02 05:42:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:42:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:42:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:42:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:42:29 Waiting for dst bucket item count to get to: 5000 curr count: 1055
2021/12/02 05:42:34 Waiting for dst bucket item count to get to: 5000 curr count: 2112
2021/12/02 05:42:39 Waiting for dst bucket item count to get to: 5000 curr count: 3557
2021/12/02 05:42:39 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:42:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:42:39 Pausing app: bucket_op_with_timer
2021/12/02 05:42:39 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:42:44 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/02 05:42:44 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/02 05:42:44 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 05:42:45 Removing node(s): [] from the cluster
2021/12/02 05:42:50 Rebalance progress: 29.16666666666666
2021/12/02 05:42:55 Rebalance progress: 29.16666666666666
2021/12/02 05:43:00 Rebalance progress: 100
2021/12/02 05:43:00 Resuming app: bucket_op_with_timer
2021/12/02 05:43:01 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:43:01 Starting bucket ops to source bucket
2021/12/02 05:43:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:43:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:43:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:43:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:43:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/02 05:43:39 Waiting for dst bucket item count to get to: 10000 curr count: 8219
2021/12/02 05:43:44 Waiting for dst bucket item count to get to: 10000 curr count: 8219
2021/12/02 05:43:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 05:43:44 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 05:43:49 Rebalance progress: 29.16666666666666
2021/12/02 05:43:54 Rebalance progress: 29.16666666666666
2021/12/02 05:43:59 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/02 05:44:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:44:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:44:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:44:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:45:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:45:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:45:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:45:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:46:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:46:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:46:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:46:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:47:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:47:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:47:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:47:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:48:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:48:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:48:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:48:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:49:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:49:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:49:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 05:49:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:61} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cd5fe9dfdee0122002f686ecd9f7f942
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
Producer node: 172.31.2.134:9300	Node UUID: 31e96232e10e91f957615aeb546c8c0b
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
Producer node: 127.0.0.1:9303	Node UUID: 731b5b538a99bfff7a8a0ff45956c5e0
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/02 05:50:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:50:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 05:50:09 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 05:50:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 05:50:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 05:50:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/02 05:50:24 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/02 05:50:24 eventingSetup.go:46: No eventing-con process running
2021/12/02 05:50:24 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/02 05:50:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:50:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 05:50:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:50:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:50:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8121
2021/12/02 05:50:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3121
2021/12/02 05:50:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (578.47s)
=== RUN   TestBucketDeleteAfterPause
2021/12/02 05:51:03 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/02 05:51:03 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/02 05:51:03 eventingSetup.go:46: No eventing-con process running
2021/12/02 05:51:03 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/02 05:51:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:51:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:51:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:51:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:51:17 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:51:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/02 05:51:23 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 05:51:28 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 05:51:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 05:51:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/02 05:51:38 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/02 05:51:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 05:51:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 605
2021/12/02 05:51:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 605
2021/12/02 05:51:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3105
2021/12/02 05:51:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:51:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:51:58 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/02 05:51:58 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:52:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/02 05:52:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/02 05:52:03 handler_test.go:716: Deleting source bucket: default
2021/12/02 05:52:10 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/02 05:52:10 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/02 05:52:15 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/02 05:52:15 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/02 05:52:15 eventingSetup.go:46: No eventing-con process running
2021/12/02 05:52:15 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/02 05:52:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:52:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 05:52:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:52:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:52:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3318
2021/12/02 05:52:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 818
2021/12/02 05:52:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (109.60s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/02 05:52:53 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 05:52:53 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/02 05:52:53 eventingSetup.go:46: No eventing-con process running
2021/12/02 05:52:53 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/02 05:52:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:52:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:53:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:53:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:53:06 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:53:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/02 05:53:12 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 05:53:17 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 05:53:22 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 05:53:27 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/02 05:53:27 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/02 05:53:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 05:54:02 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/02 05:54:03 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:54:08 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/02 05:54:08 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/02 05:57:08 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/02 05:57:08 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/02 05:57:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/02 05:57:14 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 05:57:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:57:14 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/02 05:57:19 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 05:57:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:57:19 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2731 expected 5000
2021/12/02 05:57:24 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 05:57:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:57:24 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3357 expected 5000
2021/12/02 05:57:29 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/02 05:57:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:57:29 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/02 05:57:29 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/02 05:57:29 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:57:39 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 05:57:44 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 05:57:49 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/02 05:57:49 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/02 05:57:49 eventingSetup.go:46: No eventing-con process running
2021/12/02 05:57:49 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/02 05:57:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:58:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3876
2021/12/02 05:58:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3876
2021/12/02 05:58:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:58:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:58:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.15s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/02 05:58:22 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 05:58:22 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/02 05:58:22 eventingSetup.go:46: No eventing-con process running
2021/12/02 05:58:22 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/02 05:58:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:58:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:58:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 05:58:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 05:58:35 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 05:58:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 05:58:41 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 05:58:46 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 05:58:51 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 05:58:56 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 05:58:56 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/02 05:58:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 05:59:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 05:59:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2068
2021/12/02 05:59:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2068
2021/12/02 05:59:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 05:59:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 05:59:16 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/02 05:59:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 05:59:46 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/02 05:59:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 05:59:52 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 05:59:57 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 06:00:02 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 06:00:07 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/02 06:00:07 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/02 06:00:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:00:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:00:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:00:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:00:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:00:25 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/02 06:00:25 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 06:00:35 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 06:00:40 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 06:00:45 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/02 06:00:45 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/02 06:00:45 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:00:45 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/02 06:00:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:00:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:00:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:00:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.97s)
=== RUN   TestCPPWorkerCleanup
2021/12/02 06:01:08 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 06:01:08 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/02 06:01:08 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:01:08 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/02 06:01:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:01:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:01:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:01:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:01:21 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:01:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/02 06:01:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:01:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:01:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:01:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:01:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:01:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2963
2021/12/02 06:01:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2963
2021/12/02 06:01:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:01:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:01:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:02:00 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 06:02:05 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 06:02:10 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 06:02:15 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/02 06:02:15 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/02 06:02:15 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:02:15 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/02 06:02:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:02:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:02:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:02:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:02:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3066
2021/12/02 06:02:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 566
2021/12/02 06:02:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.82s)
=== RUN   TestWithUserXattrs
2021/12/02 06:03:23 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 06:03:23 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/02 06:03:23 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:03:23 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/02 06:03:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:03:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:03:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:03:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:03:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:03:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:03:40 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:03:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/02 06:03:46 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 06:03:51 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 06:03:56 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 06:04:01 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/02 06:04:01 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/02 06:04:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:04:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:04:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:04:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:04:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:04:21 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 06:04:26 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 06:04:31 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 06:04:36 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/02 06:04:36 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/02 06:04:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:04:36 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/02 06:04:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:04:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:04:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:04:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:04:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4732
2021/12/02 06:05:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2232
2021/12/02 06:05:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.02s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/02 06:05:14 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 06:05:14 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/02 06:05:14 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:05:14 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/02 06:05:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:05:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:05:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:05:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:05:27 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:05:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/02 06:05:33 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 06:05:38 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 06:05:43 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 06:05:48 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/02 06:05:48 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/02 06:05:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:05:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:06:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 672
2021/12/02 06:06:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3172
2021/12/02 06:06:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:06:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:06:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:06:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/02 06:06:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/02 06:06:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:06:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:06:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 06:06:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:06:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:07:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 13100
2021/12/02 06:07:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 19460
2021/12/02 06:07:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 24744
2021/12/02 06:07:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/02 06:07:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:07:13 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 06:07:18 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 06:07:23 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 06:07:28 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 06:07:33 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 06:07:33 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/02 06:07:33 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:07:38 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/02 06:07:38 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/02 06:07:38 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:07:38 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/02 06:07:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:07:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 06:07:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/02 06:07:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:07:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:08:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/02 06:08:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.15s)
=== RUN   TestMetaBucketDelete
2021/12/02 06:08:16 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/02 06:08:16 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/02 06:08:16 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:08:16 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/02 06:08:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:08:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:08:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:08:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:08:29 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:08:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 06:08:35 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 06:08:40 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 06:08:45 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 06:08:50 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/02 06:08:50 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/02 06:08:50 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/02 06:08:52 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/02 06:08:57 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/02 06:08:57 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/02 06:09:12 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/02 06:09:12 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/02 06:09:12 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:09:12 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/02 06:09:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:09:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:09:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:09:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (79.20s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/02 06:09:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:09:35 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:09:35 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:09:35 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/02 06:09:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:09:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:09:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:09:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:09:48 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:09:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 06:10:08 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/02 06:10:09 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/02 06:10:24 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:10:24 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:10:39 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:10:39 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:10:39 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:10:39 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/02 06:10:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:10:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:10:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:10:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.90s)
=== RUN   TestSourceBucketDelete
2021/12/02 06:11:02 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/02 06:11:02 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/02 06:11:02 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:11:02 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/02 06:11:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:11:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:11:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:11:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:11:15 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:11:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/02 06:11:21 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 06:11:26 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 06:11:31 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 06:11:36 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/02 06:11:36 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/02 06:11:36 handler_test.go:945: Deleting source bucket: default
2021/12/02 06:11:38 handler_test.go:947: Deleted source bucket: default
2021/12/02 06:11:43 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/02 06:11:43 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/02 06:11:58 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/02 06:11:58 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/02 06:11:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:11:58 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/02 06:12:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:12:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:12:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:12:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.51s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/02 06:12:22 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:12:22 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:12:22 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:12:22 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/02 06:12:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:12:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:12:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:12:35 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:12:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 06:12:55 handler_test.go:966: Deleting source bucket: default
2021/12/02 06:12:56 handler_test.go:968: Deleted source bucket: default
2021/12/02 06:13:11 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:13:11 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:13:26 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:13:26 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:13:26 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:13:26 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/02 06:13:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:13:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:13:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:13:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.72s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/02 06:13:49 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:13:49 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:13:49 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:13:49 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/02 06:13:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:13:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:13:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:14:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:14:03 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:14:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 06:14:22 handler_test.go:988: Deleting source bucket: default
2021/12/02 06:14:24 handler_test.go:990: Deleted source bucket: default
2021/12/02 06:14:24 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/02 06:14:24 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/02 06:14:39 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:14:39 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:14:54 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/02 06:14:54 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/02 06:14:54 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:14:54 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/02 06:14:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:15:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:15:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:15:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.27s)
=== RUN   TestUndeployDuringBootstrap
2021/12/02 06:15:18 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 06:15:18 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/02 06:15:18 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:15:18 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/02 06:15:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:15:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:15:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:15:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:15:31 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:15:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 06:15:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:15:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:15:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:15:44 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/02 06:15:44 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/02 06:15:44 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/02 06:15:49 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/02 06:15:54 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/02 06:15:59 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 06:16:04 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 06:16:09 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 06:16:09 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/02 06:16:09 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:16:34 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/02 06:16:34 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/02 06:16:34 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:16:34 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/02 06:16:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:16:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:16:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:16:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:16:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:16:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.70s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/02 06:17:02 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:17:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/02 06:17:08 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 06:17:13 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 06:17:18 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 06:17:23 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/02 06:17:23 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/02 06:17:23 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/02 06:17:24 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 06:17:24 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/02 06:17:29 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/02 06:17:29 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/02 06:17:29 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:17:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/02 06:17:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:17:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:17:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:17:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.60s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/02 06:17:52 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 06:17:52 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/02 06:17:52 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:17:52 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/02 06:17:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:17:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:18:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:18:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:18:05 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:18:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/02 06:18:12 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 06:18:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 06:18:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 06:18:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/02 06:18:27 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/02 06:18:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:18:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:19:02 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 06:19:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 06:19:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 06:19:17 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 06:19:17 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/02 06:19:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:21:02 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/02 06:21:02 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/02 06:21:02 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:21:02 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/02 06:21:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:21:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:21:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 06:21:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:21:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:21:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3315
2021/12/02 06:21:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3315
2021/12/02 06:21:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.25s)
=== RUN   TestUndeployWithKVFailover
2021/12/02 06:21:45 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/02 06:21:45 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/02 06:21:45 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:21:45 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/02 06:21:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:21:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:21:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:21:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:21:58 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:21:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/02 06:21:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 06:22:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:22:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:22:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:22:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:22:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:22:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:22:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1366
2021/12/02 06:22:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3866
2021/12/02 06:22:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:22:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 06:22:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 06:22:47 Rebalance progress: 29.16666666666666
2021/12/02 06:22:52 Rebalance progress: 29.16666666666666
2021/12/02 06:22:57 Rebalance progress: 100
2021/12/02 06:22:57 Failing over: [127.0.0.1:19003]
2021/12/02 06:23:07 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/02 06:24:07 Removing node(s): [] from the cluster
2021/12/02 06:24:12 Rebalance progress: 29.16666666666666
2021/12/02 06:24:17 Rebalance progress: 29.16666666666666
2021/12/02 06:24:22 Rebalance progress: 100
2021/12/02 06:24:27 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/02 06:24:27 App: TestUndeployWithKVFailover got un-deployed
2021/12/02 06:24:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:24:32 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/02 06:24:32 App: TestUndeployWithKVFailover got un-deployed
2021/12/02 06:24:32 No eventing-con process running
2021/12/02 06:24:32 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/02 06:24:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:24:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:24:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 06:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:24:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:25:00 Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/02 06:25:05 Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/02 06:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (209.91s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/02 06:25:15 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 06:25:15 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/02 06:25:15 No eventing-con process running
2021/12/02 06:25:15 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/02 06:25:16 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:25:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:25:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:25:28 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:25:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/02 06:25:34 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 06:25:39 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 06:25:44 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 06:25:49 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/02 06:25:49 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/02 06:25:49 Starting bucket ops to source bucket
2021/12/02 06:26:10 Waiting for dst bucket item count to get to: 20000 curr count: 11278
2021/12/02 06:26:15 Waiting for dst bucket item count to get to: 20000 curr count: 16329
2021/12/02 06:26:15 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/02 06:26:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:26:18 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:26:28 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 06:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:26:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:26:39 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 06:26:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 06:26:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:26:44 Starting bucket ops to source bucket
2021/12/02 06:26:57 Waiting for dst bucket item count to get to: 10000 curr count: 4664
2021/12/02 06:27:02 Waiting for dst bucket item count to get to: 10000 curr count: 4664
2021/12/02 06:27:02 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 06:27:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:27:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:27:07 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 06:27:12 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 06:27:17 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 06:27:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/02 06:27:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/02 06:27:23 No eventing-con process running
2021/12/02 06:27:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/02 06:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:27:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 06:27:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:27:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:27:51 Waiting for dst bucket item count to get to: 0 curr count: 8617
2021/12/02 06:27:56 Waiting for dst bucket item count to get to: 0 curr count: 8617
2021/12/02 06:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.63s)
=== RUN   TestTimerOverwrite
2021/12/02 06:28:06 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 06:28:06 App: TestTimerOverwrite got un-deployed
2021/12/02 06:28:06 No eventing-con process running
2021/12/02 06:28:06 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/02 06:28:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:28:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:28:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:28:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:28:19 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:28:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/02 06:28:25 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 06:28:30 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 06:28:35 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 06:28:40 Waiting for app: TestTimerOverwrite to get deployed
2021/12/02 06:28:40 App: TestTimerOverwrite got deployed
2021/12/02 06:28:50 Starting bucket ops to source bucket
2021/12/02 06:28:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 06:29:59 Waiting for dst bucket item count to get to: 5000 curr count: 2627
2021/12/02 06:30:04 Waiting for dst bucket item count to get to: 5000 curr count: 2627
2021/12/02 06:30:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:30:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:30:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:30:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 06:30:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 06:30:30 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 06:30:35 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/02 06:30:35 App: TestTimerOverwrite got un-deployed
2021/12/02 06:30:35 No eventing-con process running
2021/12/02 06:30:35 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/02 06:30:36 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:30:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:30:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:30:58 Waiting for dst bucket item count to get to: 0 curr count: 3039
2021/12/02 06:31:03 Waiting for dst bucket item count to get to: 0 curr count: 566
2021/12/02 06:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (187.14s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/02 06:31:08 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 06:31:08 Removing node(s): [] from the cluster
2021/12/02 06:31:13 Rebalance progress: 100
2021/12/02 06:31:13 Failing over: [127.0.0.1:19003]
2021/12/02 06:31:18 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/02 06:31:18 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/02 06:31:18 No eventing-con process running
2021/12/02 06:31:18 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/02 06:31:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:31:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:31:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:31:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:31:31 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:31:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/02 06:31:32 Starting bucket ops to source bucket
2021/12/02 06:31:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:31:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:31:43 Context: Post to set retry counter, response code: 200 dump: 
2021/12/02 06:31:43 set retry counter, response &{[] }
2021/12/02 06:32:48 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/02 06:32:48 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/02 06:32:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:32:48 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/02 06:32:53 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/02 06:32:53 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/02 06:32:53 No eventing-con process running
2021/12/02 06:32:53 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/02 06:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:33:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:33:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/02 06:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:33:21 Waiting for dst bucket item count to get to: 0 curr count: 4441
2021/12/02 06:33:26 Waiting for dst bucket item count to get to: 0 curr count: 4441
2021/12/02 06:33:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:33:26 Removing node(s): [] from the cluster
2021/12/02 06:33:31 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (148.00s)
=== RUN   TestOnUpdateSrcMutation
2021/12/02 06:33:41 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 06:33:41 App: TestOnUpdateSrcMutation got un-deployed
2021/12/02 06:33:41 No eventing-con process running
2021/12/02 06:33:41 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/02 06:33:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:33:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:33:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:33:54 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:33:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/02 06:33:55 Starting bucket ops to source bucket
2021/12/02 06:34:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 06:34:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:34:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:34:27 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 06:34:32 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 06:34:37 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 06:34:42 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/02 06:34:42 App: TestOnUpdateSrcMutation got un-deployed
2021/12/02 06:34:42 No eventing-con process running
2021/12/02 06:34:43 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/02 06:34:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:34:54 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 06:34:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:34:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.80s)
=== RUN   TestOnDeleteSrcMutation
2021/12/02 06:35:16 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 06:35:16 App: TestOnDeleteSrcMutation got un-deployed
2021/12/02 06:35:16 No eventing-con process running
2021/12/02 06:35:16 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/02 06:35:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:35:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:35:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:35:29 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:35:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/02 06:35:30 Starting bucket ops to source bucket
2021/12/02 06:35:59 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:35:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:35:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:36:04 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 06:36:09 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 06:36:14 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 06:36:19 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/02 06:36:19 App: TestOnDeleteSrcMutation got un-deployed
2021/12/02 06:36:19 No eventing-con process running
2021/12/02 06:36:19 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/02 06:36:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:36:30 Waiting for dst bucket item count to get to: 0 curr count: 3623
2021/12/02 06:36:35 Waiting for dst bucket item count to get to: 0 curr count: 3623
2021/12/02 06:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:36:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.35s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/02 06:36:52 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 06:36:52 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/02 06:36:52 No eventing-con process running
2021/12/02 06:36:52 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/02 06:36:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:36:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:37:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:37:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:37:05 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:37:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 06:37:06 Starting bucket ops to source bucket
2021/12/02 06:37:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 06:37:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:37:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:37:48 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 06:37:53 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 06:37:58 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 06:38:04 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/02 06:38:04 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/02 06:38:04 No eventing-con process running
2021/12/02 06:38:04 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/02 06:38:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:38:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 06:38:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:38:17 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:38:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.67s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/02 06:38:32 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 06:38:32 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/02 06:38:32 No eventing-con process running
2021/12/02 06:38:32 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/02 06:38:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:38:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:38:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:38:45 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:38:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/02 06:38:46 Starting bucket ops to source bucket
2021/12/02 06:39:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 06:39:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:39:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:39:30 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 06:39:35 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 06:39:40 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 06:39:45 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/02 06:39:45 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/02 06:39:45 No eventing-con process running
2021/12/02 06:39:45 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/02 06:39:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:39:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 06:39:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:39:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.43s)
=== RUN   TestInterHandlerRecursion
2021/12/02 06:40:13 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:40:13 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 06:40:13 No eventing-con process running
2021/12/02 06:40:13 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/02 06:40:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:40:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:40:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:40:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:40:31 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:40:31 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 06:40:31 No eventing-con process running
2021/12/02 06:40:31 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/02 06:40:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:40:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:40:44 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:40:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/02 06:40:46 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/02 06:40:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:40:56 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:41:01 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:41:06 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:41:06 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/02 06:41:06 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:41:06 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/02 06:41:06 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/02 06:41:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:41:11 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:41:16 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:41:21 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:41:26 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:41:26 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 06:41:26 No eventing-con process running
2021/12/02 06:41:26 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/02 06:41:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:41:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:41:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:41:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:41:44 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:41:44 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 06:41:44 No eventing-con process running
2021/12/02 06:41:44 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/02 06:41:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:41:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:41:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:41:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.04s)
=== RUN   TestInterBucketRecursion
2021/12/02 06:42:02 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:42:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 06:42:03 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 06:42:03 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:42:03 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/02 06:42:08 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 06:42:13 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 06:42:18 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 06:42:23 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/02 06:42:23 App: TestInterBucketRecursion_function1 got deployed
2021/12/02 06:42:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:42:28 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:42:33 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:42:38 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:42:43 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:42:43 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/02 06:42:43 No eventing-con process running
2021/12/02 06:42:44 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/02 06:42:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:42:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:42:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:42:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:43:02 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/02 06:43:02 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/02 06:43:02 No eventing-con process running
2021/12/02 06:43:02 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/02 06:43:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:43:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:43:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.73s)
=== RUN   TestLargeHandler
2021/12/02 06:43:15 Changed force_compress value to false
2021/12/02 06:43:15 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:43:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/02 06:43:15 Changed force_compress value to true
2021/12/02 06:43:15 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:43:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/02 06:43:22 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 06:43:27 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 06:43:32 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 06:43:37 Waiting for app: TestLargeHandler_function to get deployed
2021/12/02 06:43:37 App: TestLargeHandler_function got deployed
2021/12/02 06:43:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:43:42 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 06:43:47 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 06:43:52 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 06:43:57 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/02 06:43:57 App: TestLargeHandler_function got un-deployed
2021/12/02 06:43:57 No eventing-con process running
2021/12/02 06:43:57 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/02 06:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:44:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:44:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.50s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/02 06:44:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:44:20 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 06:44:20 No eventing-con process running
2021/12/02 06:44:20 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/02 06:44:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:44:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:44:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:44:39 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:44:39 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 06:44:39 No eventing-con process running
2021/12/02 06:44:39 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/02 06:44:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:44:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:44:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:44:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:44:52 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:44:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/02 06:44:53 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/02 06:44:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:45:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:45:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:45:13 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/02 06:45:13 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/02 06:45:13 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:45:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/02 06:45:14 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/02 06:45:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 06:45:24 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 06:45:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 06:45:34 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/02 06:45:34 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/02 06:45:34 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/02 06:45:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:45:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:45:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:45:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:45:55 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/02 06:45:55 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/02 06:45:55 No eventing-con process running
2021/12/02 06:45:55 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/02 06:45:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:46:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:46:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:46:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:46:14 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:46:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:46:24 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:46:29 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/02 06:46:29 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/02 06:46:29 No eventing-con process running
2021/12/02 06:46:29 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/02 06:46:30 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:46:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:46:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.97s)
=== RUN   TestAllowInterBucketRecursion
2021/12/02 06:46:47 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:46:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 06:46:48 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 06:46:53 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 06:46:58 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 06:47:03 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 06:47:08 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/02 06:47:08 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/02 06:47:08 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:47:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/02 06:47:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/02 06:47:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 06:47:20 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 06:47:25 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 06:47:30 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/02 06:47:30 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/02 06:47:30 TestAllowInterBucketRecursion_function2 is deployed
2021/12/02 06:47:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:47:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:47:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:47:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:47:50 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/02 06:47:50 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/02 06:47:50 No eventing-con process running
2021/12/02 06:47:50 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/02 06:47:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:47:59 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:48:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:48:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:48:09 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 06:48:14 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 06:48:19 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 06:48:24 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/02 06:48:24 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/02 06:48:24 No eventing-con process running
2021/12/02 06:48:24 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/02 06:48:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:48:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:48:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:48:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.04s)
=== RUN   TestInterBucketRecursion2
2021/12/02 06:48:43 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:48:50 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:48:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:49:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:49:07 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:49:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 06:49:13 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 06:49:18 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 06:49:23 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 06:49:28 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/02 06:49:28 App: TestInterBucketRecursion2_function1 got deployed
2021/12/02 06:49:28 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:49:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/02 06:49:34 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 06:49:39 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 06:49:44 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 06:49:49 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/02 06:49:49 App: TestInterBucketRecursion2_function2 got deployed
2021/12/02 06:49:49 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:49:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/02 06:49:55 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 06:50:00 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 06:50:05 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 06:50:10 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/02 06:50:10 App: TestInterBucketRecursion2_function3 got deployed
2021/12/02 06:50:10 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:50:10 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/02 06:50:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:50:16 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 06:50:21 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 06:50:26 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 06:50:31 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/02 06:50:31 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/02 06:50:31 No eventing-con process running
2021/12/02 06:50:31 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/02 06:50:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:50:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:50:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:50:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:50:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:50:49 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 06:50:54 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 06:50:59 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 06:51:04 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/02 06:51:04 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/02 06:51:04 No eventing-con process running
2021/12/02 06:51:04 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/02 06:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:51:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:51:13 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:51:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:51:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 06:51:23 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 06:51:28 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 06:51:33 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 06:51:38 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/02 06:51:38 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/02 06:51:38 No eventing-con process running
2021/12/02 06:51:38 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/02 06:51:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:51:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:51:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:51:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:51:56 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/02 06:51:56 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/02 06:51:56 No eventing-con process running
2021/12/02 06:51:56 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/02 06:51:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:52:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.78s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/02 06:52:16 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 06:52:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/02 06:52:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:52:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:52:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:53:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:53:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:53:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:53:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:54:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:54:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:54:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:54:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:55:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:55:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:55:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:55:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:56:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:56:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:56:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:56:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:57:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:57:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:57:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/02 06:57:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:57} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/02 06:58:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:58:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 06:58:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 06:58:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/02 06:58:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:58:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 06:58:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 06:58:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/02 06:58:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:58:25 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/02 06:58:25 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/02 06:58:25 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:58:25 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/02 06:58:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:58:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:58:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:58:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:58:43 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 06:58:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 06:58:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 06:58:54 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 06:58:59 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 06:59:04 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/02 06:59:04 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 31e96232e10e91f957615aeb546c8c0b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 731b5b538a99bfff7a8a0ff45956c5e0
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: cd5fe9dfdee0122002f686ecd9f7f942
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/02 06:59:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:59:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 06:59:10 Rebalance progress: 29.16666666666666
2021/12/02 06:59:15 Rebalance progress: 29.16666666666666
2021/12/02 06:59:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cd5fe9dfdee0122002f686ecd9f7f942
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 31e96232e10e91f957615aeb546c8c0b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 731b5b538a99bfff7a8a0ff45956c5e0
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/02 06:59:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:59:20 handler_test.go:1674: Deleting source bucket: default
2021/12/02 06:59:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 06:59:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 06:59:27 Rebalance progress: 29.16666666666666
2021/12/02 06:59:32 Rebalance progress: 29.16666666666666
2021/12/02 06:59:37 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 903
	vb list dump: [0-8, 10-13, 15-18, 20-21, 23-40, 42-52, 54-57, 59-62, 65-76, 78-83, 85-95, 97-100, 102-118, 120-134, 136-145, 147-166, 168-170, 172-185, 187-188, 190-193, 195-200, 202, 206-226, 229, 231-239, 241-250, 252-253, 255-259, 261, 263-273, 275, 277-283, 285-291, 293-307, 309, 311-326, 329-338, 340-345, 347-350, 352-356, 359-364, 366, 368-369, 371-406, 408-417, 419-430, 432-434, 436-446, 449-450, 452-456, 458-459, 461-465, 467-471, 473-481, 483-485, 488-489, 491, 493-495, 497-501, 503-508, 512-573, 575-598, 600-604, 606-607, 609-623, 625-635, 638-653, 655-657, 659, 661-674, 676, 678-690, 692-731, 733, 735-740, 743-763, 765-766, 768-772, 774-777, 779-780, 782-787, 791-810, 813-825, 827-836, 838-844, 847, 849, 851-866, 868-872, 874-893, 895-898, 901-902, 904-912, 914-921, 923-941, 943-954, 957-965, 967-978, 980-986, 988-989, 991-1000, 1002, 1004-1010, 1012-1023]
	stream status: stopped
	len: 121
	vb list dump: [9, 14, 19, 22, 41, 53, 58, 63-64, 77, 84, 96, 101, 119, 135, 146, 167, 171, 186, 189, 194, 201, 203-205, 227-228, 230, 240, 251, 254, 260, 262, 274, 276, 284, 292, 308, 310, 327-328, 339, 346, 351, 357-358, 365, 367, 370, 407, 418, 431, 435, 447-448, 451, 457, 460, 466, 472, 482, 486-487, 490, 492, 496, 502, 509-511, 574, 599, 605, 608, 624, 636-637, 654, 658, 660, 675, 677, 691, 732, 734, 741-742, 764, 767, 773, 778, 781, 788-790, 811-812, 826, 837, 845-846, 848, 850, 867, 873, 894, 899-900, 903, 913, 922, 942, 955-956, 966, 979, 987, 990, 1001, 1003, 1011]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cd5fe9dfdee0122002f686ecd9f7f942
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 98
	v: [229, 231-239, 241-250, 252-253, 255-259, 261, 263-273, 275, 277-283, 285-291, 293-307, 309, 311-326, 329-338, 340-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-8, 10-13, 15-18, 20-21, 23-40, 42-52, 54-57, 59-62, 65-76, 78-83, 85-95, 97-100, 102-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-118, 120-134, 136-145, 147-166, 168-170, 172-185, 187-188, 190-193, 195-200, 202, 206-226]
Producer node: 172.31.2.134:9300	Node UUID: 31e96232e10e91f957615aeb546c8c0b
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [797-810, 813-825, 827-836, 838-844, 847, 849, 851-866, 868-872, 874-893, 895-898, 901-902, 904-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-912, 914-921, 923-941, 943-954, 957-965, 967-978, 980-986, 988-989, 991-1000, 1002, 1004-1010, 1012-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-690, 692-731, 733, 735-740, 743-763, 765-766, 768-772, 774-777, 779-780, 782-787, 791-796]
Producer node: 127.0.0.1:9303	Node UUID: 731b5b538a99bfff7a8a0ff45956c5e0
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-345, 347-350, 352-356, 359-364, 366, 368-369, 371-406, 408-417, 419-430, 432-434, 436-446, 449-450, 452-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [456, 458-459, 461-465, 467-471, 473-481, 483-485, 488-489, 491, 493-495, 497-501, 503-508, 512-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-573, 575-598, 600-604, 606-607, 609-623, 625-635, 638-653, 655-657, 659, 661-674, 676, 678-682]


2021/12/02 06:59:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 06:59:42 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/02 06:59:42 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/02 06:59:47 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/02 06:59:47 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/02 06:59:47 eventingSetup.go:46: No eventing-con process running
2021/12/02 06:59:47 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/02 06:59:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 06:59:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 06:59:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:00:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:00:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/02 07:00:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 07:00:06 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (477.56s)
=== RUN   TestN1QLRecursion
2021/12/02 07:00:09 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 07:00:09 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/02 07:00:09 Removing node(s): [] from the cluster
2021/12/02 07:00:14 Rebalance progress: 100
2021/12/02 07:00:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:00:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:00:37 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:00:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 07:00:38 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 07:00:43 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 07:00:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 07:00:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 07:00:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 07:01:04 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/02 07:01:04 App: TestN1QLRecursion_function1 got deployed
2021/12/02 07:01:04 TestN1QLRecursion_function1 is deployed
2021/12/02 07:01:04 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:01:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/02 07:01:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/02 07:01:10 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 07:01:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 07:01:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 07:01:25 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/02 07:01:25 App: TestN1QLRecursion_function3 got deployed
2021/12/02 07:01:25 TestN1QLRecursion_function3 is deployed
2021/12/02 07:01:25 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:01:25 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/02 07:01:25 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/02 07:01:25 Success: TestN1QLRecursion_function2 is not deployed
2021/12/02 07:01:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:01:30 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 07:01:35 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 07:01:40 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 07:01:45 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/02 07:01:45 App: TestN1QLRecursion_function1 got un-deployed
2021/12/02 07:01:45 No eventing-con process running
2021/12/02 07:01:45 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/02 07:01:47 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:01:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:01:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:01:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:01:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:02:04 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 07:02:09 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 07:02:14 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 07:02:19 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/02 07:02:19 App: TestN1QLRecursion_function3 got un-deployed
2021/12/02 07:02:19 No eventing-con process running
2021/12/02 07:02:19 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/02 07:02:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:02:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:02:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:02:34 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 07:02:39 Rebalance progress: 100
--- PASS: TestN1QLRecursion (150.66s)
=== RUN   TestN1QLAllowRecursion
2021/12/02 07:02:39 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 07:02:40 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/02 07:02:40 Removing node(s): [] from the cluster
2021/12/02 07:02:45 Rebalance progress: 100
2021/12/02 07:02:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:02:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:03:03 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:03:08 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:03:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/02 07:03:09 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/02 07:03:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 07:03:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 07:03:24 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 07:03:29 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/02 07:03:29 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/02 07:03:29 TestN1QLAllowRecursion_function1 is deployed
2021/12/02 07:03:29 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:03:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/02 07:03:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/02 07:03:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 07:03:40 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 07:03:45 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 07:03:50 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/02 07:03:50 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/02 07:03:50 TestN1QLAllowRecursion_function3 is deployed
2021/12/02 07:03:50 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:03:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/02 07:03:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/02 07:03:56 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 07:04:01 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 07:04:06 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 07:04:11 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/02 07:04:11 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/02 07:04:11 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/02 07:04:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:04:16 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 07:04:21 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 07:04:26 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 07:04:31 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/02 07:04:31 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/02 07:04:31 No eventing-con process running
2021/12/02 07:04:32 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/02 07:04:33 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:04:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:04:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:04:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:04:50 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 07:04:55 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 07:05:00 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 07:05:05 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/02 07:05:05 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/02 07:05:05 No eventing-con process running
2021/12/02 07:05:06 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/02 07:05:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:05:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:05:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:05:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:05:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:05:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 07:05:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 07:05:35 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 07:05:40 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/02 07:05:40 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/02 07:05:40 No eventing-con process running
2021/12/02 07:05:40 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/02 07:05:42 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:05:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:05:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:05:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:05:55 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 07:06:00 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.67s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/02 07:06:00 Sleeping for some time
2021/12/02 07:06:10 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 07:06:10 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/02 07:06:10 No eventing-con process running
2021/12/02 07:06:10 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/02 07:06:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:06:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:06:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:06:24 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:06:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 07:06:25 Starting bucket ops to source bucket
2021/12/02 07:07:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 07:07:19 Waiting for dst bucket item count to get to: 5000 curr count: 2127
2021/12/02 07:07:24 Waiting for dst bucket item count to get to: 5000 curr count: 2127
2021/12/02 07:07:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 07:07:24 Starting bucket ops to source bucket
2021/12/02 07:07:25 Starting bucket ops to source bucket
2021/12/02 07:08:31 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/02 07:08:31 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/02 07:08:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:08:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:08:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 07:08:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 07:08:46 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 07:08:51 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/02 07:08:51 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/02 07:08:51 No eventing-con process running
2021/12/02 07:08:51 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/02 07:08:53 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:09:03 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:09:08 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/02 07:09:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:09:10 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:09:20 Waiting for dst bucket item count to get to: 0 curr count: 4147
2021/12/02 07:09:25 Waiting for dst bucket item count to get to: 0 curr count: 4147
2021/12/02 07:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.94s)
=== RUN   TestTimerOverWriteSameReference
2021/12/02 07:09:30 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 07:09:30 App: TestTimerOverWriteSameReference got un-deployed
2021/12/02 07:09:30 No eventing-con process running
2021/12/02 07:09:30 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/02 07:09:32 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:09:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:09:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:09:44 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:09:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/02 07:09:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 07:09:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 07:10:00 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 07:10:05 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/02 07:10:05 App: TestTimerOverWriteSameReference got deployed
2021/12/02 07:10:15 Starting bucket ops to source bucket
2021/12/02 07:11:16 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/02 07:11:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:11:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:11:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 07:11:36 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 07:11:41 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 07:11:46 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/02 07:11:46 App: TestTimerOverWriteSameReference got un-deployed
2021/12/02 07:11:46 No eventing-con process running
2021/12/02 07:11:46 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/02 07:11:48 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:11:58 Waiting for dst bucket item count to get to: 0 curr count: 674
2021/12/02 07:12:03 Waiting for dst bucket item count to get to: 0 curr count: 174
2021/12/02 07:12:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:12:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/02 07:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.98s)
=== RUN   TestCancelTimerBucketop
2021/12/02 07:12:20 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 07:12:20 App: TestCancelTimerBucketop got un-deployed
2021/12/02 07:12:20 No eventing-con process running
2021/12/02 07:12:20 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/02 07:12:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:12:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:12:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:12:34 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:12:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/02 07:12:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 07:12:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 07:12:50 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 07:12:55 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/02 07:12:55 App: TestCancelTimerBucketop got deployed
2021/12/02 07:12:55 Starting bucket ops to source bucket
2021/12/02 07:13:02 Starting bucket ops to source bucket
2021/12/02 07:14:04 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 07:14:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:14:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:14:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 07:14:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 07:14:20 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 07:14:25 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/02 07:14:25 App: TestCancelTimerBucketop got un-deployed
2021/12/02 07:14:25 No eventing-con process running
2021/12/02 07:14:25 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/02 07:14:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:14:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:14:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:14:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:14:49 Waiting for dst bucket item count to get to: 0 curr count: 849
2021/12/02 07:14:54 Waiting for dst bucket item count to get to: 0 curr count: 349
2021/12/02 07:14:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.63s)
=== RUN   TestJSExpiryDate
2021/12/02 07:14:59 Starting bucket ops to source bucket
2021/12/02 07:15:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 07:15:17 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/02 07:15:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/02 07:15:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:15:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:15:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:15:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 07:15:53 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 07:15:58 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 07:16:03 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/02 07:16:03 App: TestJSExpiryDate got un-deployed
2021/12/02 07:16:03 No eventing-con process running
2021/12/02 07:16:03 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/02 07:16:05 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:16:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:16:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.89s)
=== RUN   TestBinaryDoc
2021/12/02 07:16:22 Starting bucket ops to source bucket
2021/12/02 07:16:23 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:16:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/02 07:16:24 Starting bucket ops to source bucket
2021/12/02 07:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:16:24 Deleting source bucket docs now
2021/12/02 07:16:24 Starting bucket ops to source bucket
2021/12/02 07:17:01 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/02 07:17:06 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/02 07:17:06 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/02 07:17:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:17:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:17:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 07:17:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 07:17:21 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 07:17:26 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/02 07:17:26 App: TestBinaryDoc got un-deployed
2021/12/02 07:17:26 No eventing-con process running
2021/12/02 07:17:26 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/02 07:17:28 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:17:38 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/02 07:17:43 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/02 07:17:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:17:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:17:55 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/02 07:17:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.24s)
=== RUN   TestConstantBindings
2021/12/02 07:17:55 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:17:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/02 07:17:56 Starting bucket ops to source bucket
2021/12/02 07:18:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/02 07:18:06 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/02 07:18:11 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/02 07:18:16 Waiting for dst bucket item count to get to: 6000 curr count: 2028
2021/12/02 07:18:21 Waiting for dst bucket item count to get to: 6000 curr count: 2028
2021/12/02 07:18:22 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/02 07:18:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:18:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:18:27 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 07:18:32 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 07:18:37 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 07:18:42 Waiting for app: TestConstantBindings to get un-deployed
2021/12/02 07:18:42 App: TestConstantBindings got un-deployed
2021/12/02 07:18:42 No eventing-con process running
2021/12/02 07:18:42 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/02 07:18:44 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:18:54 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:18:59 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 07:18:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:19:00 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:19:10 Waiting for dst bucket item count to get to: 0 curr count: 4571
2021/12/02 07:19:15 Waiting for dst bucket item count to get to: 0 curr count: 4571
2021/12/02 07:19:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.76s)
=== RUN   TestCodeInsight
2021/12/02 07:19:20 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 07:19:20 App: TestCodeInsight got un-deployed
2021/12/02 07:19:20 No eventing-con process running
2021/12/02 07:19:20 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/02 07:19:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:19:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:19:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:19:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:19:34 Starting bucket ops to source bucket
2021/12/02 07:19:35 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:19:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/02 07:19:41 Waiting for app: TestCodeInsight to get deployed
2021/12/02 07:19:46 Waiting for app: TestCodeInsight to get deployed
2021/12/02 07:19:51 Waiting for app: TestCodeInsight to get deployed
2021/12/02 07:19:56 Waiting for app: TestCodeInsight to get deployed
2021/12/02 07:19:56 App: TestCodeInsight got deployed
2021/12/02 07:20:01 Waiting for dst bucket item count to get to: 1000 curr count: 388
2021/12/02 07:20:01 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 07:20:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:20:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 07:20:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 07:20:16 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 07:20:21 Waiting for app: TestCodeInsight to get un-deployed
2021/12/02 07:20:21 App: TestCodeInsight got un-deployed
2021/12/02 07:20:21 No eventing-con process running
2021/12/02 07:20:21 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/02 07:20:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:20:33 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:20:38 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 07:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:20:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:20:50 Waiting for dst bucket item count to get to: 0 curr count: 571
2021/12/02 07:20:55 Waiting for dst bucket item count to get to: 0 curr count: 571
2021/12/02 07:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.48s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13159.294s

Functional tests took 3 hours, 39 minutes and 24 seconds.

N1ql tests

2021/12/02 07:21:04 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/02 07:21:14 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/02 07:21:14 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/02 07:21:14 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 07:21:15 Removing node(s): [] from the cluster
2021/12/02 07:21:20 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/02 07:21:25 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/02 07:21:25 App: TestRecursiveMutationN1QL got un-deployed
2021/12/02 07:21:25 No eventing-con process running
2021/12/02 07:21:25 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/02 07:21:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:21:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:21:34 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:21:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:21:39 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:21:39 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.77s)
=== RUN   TestFlexReset1
2021/12/02 07:21:44 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 07:21:44 App: TestFlexReset1 got un-deployed
2021/12/02 07:21:44 No eventing-con process running
2021/12/02 07:21:44 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/02 07:21:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:21:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:21:52 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:21:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:21:57 Starting bucket ops to source bucket
2021/12/02 07:21:58 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:21:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/02 07:22:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 07:22:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 07:22:14 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 07:22:19 Waiting for app: TestFlexReset1 to get deployed
2021/12/02 07:22:19 App: TestFlexReset1 got deployed
2021/12/02 07:22:24 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/02 07:22:24 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 07:22:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:22:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:22:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 07:22:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 07:22:39 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 07:22:44 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/02 07:22:44 App: TestFlexReset1 got un-deployed
2021/12/02 07:22:44 No eventing-con process running
2021/12/02 07:22:44 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/02 07:22:46 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:22:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:22:58 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:23:08 Waiting for dst bucket item count to get to: 0 curr count: 711
2021/12/02 07:23:13 Waiting for dst bucket item count to get to: 0 curr count: 211
2021/12/02 07:23:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.46s)
=== RUN   TestFlexReset2
2021/12/02 07:23:18 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 07:23:18 App: TestFlexReset2 got un-deployed
2021/12/02 07:23:18 No eventing-con process running
2021/12/02 07:23:18 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/02 07:23:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:23:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:23:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:23:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:23:32 Starting bucket ops to source bucket
2021/12/02 07:23:32 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:23:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/02 07:23:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 07:23:43 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 07:23:48 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 07:23:53 Waiting for app: TestFlexReset2 to get deployed
2021/12/02 07:23:53 App: TestFlexReset2 got deployed
2021/12/02 07:23:59 Waiting for dst bucket item count to get to: 1000 curr count: 696
2021/12/02 07:24:04 Waiting for dst bucket item count to get to: 1000 curr count: 696
2021/12/02 07:24:04 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/02 07:24:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:24:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:24:09 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 07:24:14 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 07:24:19 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 07:24:24 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/02 07:24:24 App: TestFlexReset2 got un-deployed
2021/12/02 07:24:24 No eventing-con process running
2021/12/02 07:24:24 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/02 07:24:25 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:24:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:24:37 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:24:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/02 07:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (94.28s)
=== RUN   TestN1QLLabelledBreak
2021/12/02 07:24:52 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 07:24:52 App: TestN1QLLabelledBreak got un-deployed
2021/12/02 07:24:52 No eventing-con process running
2021/12/02 07:24:52 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/02 07:24:54 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:24:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:25:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:25:06 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:25:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/02 07:25:12 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 07:25:17 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 07:25:22 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 07:25:27 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/02 07:25:27 App: TestN1QLLabelledBreak got deployed
2021/12/02 07:25:27 Starting bucket ops to source bucket
2021/12/02 07:25:36 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/02 07:25:41 Waiting for dst bucket item count to get to: 10000 curr count: 143
2021/12/02 07:25:46 Waiting for dst bucket item count to get to: 10000 curr count: 143
2021/12/02 07:25:51 Waiting for dst bucket item count to get to: 10000 curr count: 3352
2021/12/02 07:25:56 Waiting for dst bucket item count to get to: 10000 curr count: 3352
2021/12/02 07:26:01 Waiting for dst bucket item count to get to: 10000 curr count: 7389
2021/12/02 07:26:06 Waiting for dst bucket item count to get to: 10000 curr count: 7389
2021/12/02 07:26:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 07:26:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:26:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:26:12 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 07:26:17 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 07:26:22 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 07:26:27 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/02 07:26:27 App: TestN1QLLabelledBreak got un-deployed
2021/12/02 07:26:27 No eventing-con process running
2021/12/02 07:26:27 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/02 07:26:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:26:39 Waiting for dst bucket item count to get to: 0 curr count: 4149
2021/12/02 07:26:44 Waiting for dst bucket item count to get to: 0 curr count: 1649
2021/12/02 07:26:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:26:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:26:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 07:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.12s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/02 07:27:00 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 07:27:00 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/02 07:27:00 No eventing-con process running
2021/12/02 07:27:00 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/02 07:27:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:27:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:27:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:27:14 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:27:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/02 07:27:20 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 07:27:25 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 07:27:30 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 07:27:35 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/02 07:27:35 App: TestN1QLUnlabelledBreak got deployed
2021/12/02 07:27:35 Starting bucket ops to source bucket
2021/12/02 07:27:45 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/02 07:27:50 Waiting for dst bucket item count to get to: 10000 curr count: 625
2021/12/02 07:27:55 Waiting for dst bucket item count to get to: 10000 curr count: 625
2021/12/02 07:28:00 Waiting for dst bucket item count to get to: 10000 curr count: 5918
2021/12/02 07:28:05 Waiting for dst bucket item count to get to: 10000 curr count: 5918
2021/12/02 07:28:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 07:28:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:28:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:28:10 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 07:28:15 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 07:28:20 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 07:28:25 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/02 07:28:25 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/02 07:28:25 No eventing-con process running
2021/12/02 07:28:25 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/02 07:28:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:28:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 07:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:28:39 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:28:49 Waiting for dst bucket item count to get to: 0 curr count: 9090
2021/12/02 07:28:54 Waiting for dst bucket item count to get to: 0 curr count: 4090
2021/12/02 07:28:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.36s)
=== RUN   TestN1QLThrowStatement
2021/12/02 07:28:59 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 07:28:59 App: TestN1QLThrowStatement got un-deployed
2021/12/02 07:28:59 No eventing-con process running
2021/12/02 07:28:59 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/02 07:29:01 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:29:07 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:29:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:29:12 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:29:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/02 07:29:19 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 07:29:24 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 07:29:29 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 07:29:34 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/02 07:29:34 App: TestN1QLThrowStatement got deployed
2021/12/02 07:29:34 Starting bucket ops to source bucket
2021/12/02 07:29:43 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/02 07:29:48 Waiting for dst bucket item count to get to: 10000 curr count: 250
2021/12/02 07:29:53 Waiting for dst bucket item count to get to: 10000 curr count: 792
2021/12/02 07:29:58 Waiting for dst bucket item count to get to: 10000 curr count: 2702
2021/12/02 07:30:03 Waiting for dst bucket item count to get to: 10000 curr count: 4756
2021/12/02 07:30:08 Waiting for dst bucket item count to get to: 10000 curr count: 6777
2021/12/02 07:30:13 Waiting for dst bucket item count to get to: 10000 curr count: 8774
2021/12/02 07:30:18 Waiting for dst bucket item count to get to: 10000 curr count: 9593
2021/12/02 07:30:18 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/02 07:30:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:30:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:30:23 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 07:30:28 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 07:30:33 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 07:30:38 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/02 07:30:38 App: TestN1QLThrowStatement got un-deployed
2021/12/02 07:30:38 No eventing-con process running
2021/12/02 07:30:38 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/02 07:30:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:30:50 Waiting for dst bucket item count to get to: 0 curr count: 3315
2021/12/02 07:30:55 Waiting for dst bucket item count to get to: 0 curr count: 3315
2021/12/02 07:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:30:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:31:07 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/02 07:31:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.33s)
=== RUN   TestN1QLNestedForLoop
2021/12/02 07:31:12 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 07:31:12 App: TestN1QLNestedForLoop got un-deployed
2021/12/02 07:31:12 No eventing-con process running
2021/12/02 07:31:12 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/02 07:31:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:31:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:31:21 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:31:26 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:31:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/02 07:31:32 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 07:31:37 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 07:31:42 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 07:31:47 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/02 07:31:47 App: TestN1QLNestedForLoop got deployed
2021/12/02 07:31:47 Starting bucket ops to source bucket
2021/12/02 07:31:57 Waiting for dst bucket item count to get to: 5000 curr count: 68
2021/12/02 07:32:02 Waiting for dst bucket item count to get to: 5000 curr count: 68
2021/12/02 07:32:07 Waiting for dst bucket item count to get to: 5000 curr count: 68
2021/12/02 07:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 1840
2021/12/02 07:32:17 Waiting for dst bucket item count to get to: 5000 curr count: 1840
2021/12/02 07:32:22 Waiting for dst bucket item count to get to: 5000 curr count: 4185
2021/12/02 07:32:27 Waiting for dst bucket item count to get to: 5000 curr count: 4185
2021/12/02 07:32:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 07:32:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:32:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:32:32 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 07:32:37 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 07:32:42 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 07:32:47 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/02 07:32:47 App: TestN1QLNestedForLoop got un-deployed
2021/12/02 07:32:47 No eventing-con process running
2021/12/02 07:32:47 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/02 07:32:49 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:32:59 Waiting for dst bucket item count to get to: 0 curr count: 4606
2021/12/02 07:33:04 Waiting for dst bucket item count to get to: 0 curr count: 2106
2021/12/02 07:33:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:33:06 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:33:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 07:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.46s)
=== RUN   TestN1QLAutoClose
2021/12/02 07:33:21 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 07:33:21 App: TestN1QLAutoClose got un-deployed
2021/12/02 07:33:21 No eventing-con process running
2021/12/02 07:33:21 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/02 07:33:22 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:33:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:33:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:33:34 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:33:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/02 07:33:40 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 07:33:45 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 07:33:50 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 07:33:55 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/02 07:33:55 App: TestN1QLAutoClose got deployed
2021/12/02 07:33:55 Starting bucket ops to source bucket
2021/12/02 07:34:01 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/02 07:34:06 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/02 07:34:11 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/02 07:34:16 Waiting for dst bucket item count to get to: 500 curr count: 27
2021/12/02 07:34:21 Waiting for dst bucket item count to get to: 500 curr count: 27
2021/12/02 07:34:26 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/02 07:34:31 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/02 07:34:36 Waiting for dst bucket item count to get to: 500 curr count: 70
2021/12/02 07:34:41 Waiting for dst bucket item count to get to: 500 curr count: 70
2021/12/02 07:34:46 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/02 07:34:51 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/02 07:34:56 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/02 07:35:01 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/02 07:35:06 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/02 07:35:11 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/02 07:35:16 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/02 07:35:21 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/02 07:35:26 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/02 07:35:31 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/02 07:35:36 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/02 07:35:41 Waiting for dst bucket item count to get to: 500 curr count: 200
2021/12/02 07:35:46 Waiting for dst bucket item count to get to: 500 curr count: 200
2021/12/02 07:35:51 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/02 07:35:56 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/02 07:36:01 Waiting for dst bucket item count to get to: 500 curr count: 241
2021/12/02 07:36:06 Waiting for dst bucket item count to get to: 500 curr count: 241
2021/12/02 07:36:11 Waiting for dst bucket item count to get to: 500 curr count: 262
2021/12/02 07:36:16 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/02 07:36:21 Waiting for dst bucket item count to get to: 500 curr count: 282
2021/12/02 07:36:26 Waiting for dst bucket item count to get to: 500 curr count: 295
2021/12/02 07:36:31 Waiting for dst bucket item count to get to: 500 curr count: 303
2021/12/02 07:36:36 Waiting for dst bucket item count to get to: 500 curr count: 316
2021/12/02 07:36:41 Waiting for dst bucket item count to get to: 500 curr count: 325
2021/12/02 07:36:46 Waiting for dst bucket item count to get to: 500 curr count: 337
2021/12/02 07:36:51 Waiting for dst bucket item count to get to: 500 curr count: 346
2021/12/02 07:36:56 Waiting for dst bucket item count to get to: 500 curr count: 357
2021/12/02 07:37:01 Waiting for dst bucket item count to get to: 500 curr count: 367
2021/12/02 07:37:06 Waiting for dst bucket item count to get to: 500 curr count: 376
2021/12/02 07:37:11 Waiting for dst bucket item count to get to: 500 curr count: 386
2021/12/02 07:37:16 Waiting for dst bucket item count to get to: 500 curr count: 397
2021/12/02 07:37:21 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/02 07:37:26 Waiting for dst bucket item count to get to: 500 curr count: 420
2021/12/02 07:37:31 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/02 07:37:36 Waiting for dst bucket item count to get to: 500 curr count: 440
2021/12/02 07:37:41 Waiting for dst bucket item count to get to: 500 curr count: 451
2021/12/02 07:37:46 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/02 07:37:51 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/02 07:37:56 Waiting for dst bucket item count to get to: 500 curr count: 493
2021/12/02 07:38:01 Waiting for dst bucket item count to get to: 500 curr count: 493
2021/12/02 07:38:01 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/02 07:38:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:38:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:38:07 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 07:38:12 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 07:38:17 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 07:38:22 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/02 07:38:22 App: TestN1QLAutoClose got un-deployed
2021/12/02 07:38:22 No eventing-con process running
2021/12/02 07:38:22 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/02 07:38:23 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:38:33 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/02 07:38:38 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/02 07:38:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:38:40 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:38:50 Waiting for dst bucket item count to get to: 0 curr count: 356
2021/12/02 07:38:55 Waiting for dst bucket item count to get to: 0 curr count: 356
2021/12/02 07:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.70s)
=== RUN   TestN1QLPosParams
2021/12/02 07:39:00 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 07:39:00 App: TestN1QLPosParams got un-deployed
2021/12/02 07:39:00 No eventing-con process running
2021/12/02 07:39:00 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/02 07:39:02 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:39:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:39:09 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:39:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:39:14 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:39:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/02 07:39:20 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 07:39:25 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 07:39:30 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 07:39:35 Waiting for app: TestN1QLPosParams to get deployed
2021/12/02 07:39:35 App: TestN1QLPosParams got deployed
2021/12/02 07:39:35 Starting bucket ops to source bucket
2021/12/02 07:39:45 Waiting for dst bucket item count to get to: 5000 curr count: 325
2021/12/02 07:39:50 Waiting for dst bucket item count to get to: 5000 curr count: 1005
2021/12/02 07:39:50 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 07:39:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:39:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:39:55 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 07:40:00 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 07:40:05 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 07:40:10 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/02 07:40:10 App: TestN1QLPosParams got un-deployed
2021/12/02 07:40:10 No eventing-con process running
2021/12/02 07:40:10 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/02 07:40:12 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:40:22 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 07:40:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 07:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:40:29 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:40:39 Waiting for dst bucket item count to get to: 0 curr count: 4412
2021/12/02 07:40:44 Waiting for dst bucket item count to get to: 0 curr count: 1912
2021/12/02 07:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.41s)
=== RUN   TestN1QLExhaustConnPool
2021/12/02 07:40:49 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 07:40:49 App: TestN1QLExhaustConnPool got un-deployed
2021/12/02 07:40:49 No eventing-con process running
2021/12/02 07:40:49 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/02 07:40:51 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:40:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:41:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:41:02 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:41:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/02 07:41:08 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 07:41:14 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 07:41:19 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 07:41:24 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/02 07:41:24 App: TestN1QLExhaustConnPool got deployed
2021/12/02 07:41:24 Starting bucket ops to source bucket
2021/12/02 07:41:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/02 07:41:34 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/02 07:41:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/02 07:41:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:41:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:41:39 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 07:41:44 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 07:41:49 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 07:41:54 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/02 07:41:54 App: TestN1QLExhaustConnPool got un-deployed
2021/12/02 07:41:54 No eventing-con process running
2021/12/02 07:41:54 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/02 07:41:56 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:42:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/02 07:42:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:42:08 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:42:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/02 07:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (93.89s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/02 07:42:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 07:42:23 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/02 07:42:23 No eventing-con process running
2021/12/02 07:42:23 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/02 07:42:24 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:42:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:42:31 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:42:36 Starting bucket ops to source bucket
2021/12/02 07:42:54 Starting bucket ops to source bucket
2021/12/02 07:43:11 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 7738
2021/12/02 07:43:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/02 07:43:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 07:43:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 07:43:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 07:43:33 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/02 07:43:33 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/02 07:43:33 Starting bucket ops to source bucket
2021/12/02 07:43:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:43:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:43:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:43:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:43:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:44:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:44:08 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:44:13 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:44:18 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:44:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/02 07:44:23 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/02 07:44:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:44:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:44:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 07:44:33 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 07:44:38 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 07:44:43 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/02 07:44:43 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/02 07:44:43 No eventing-con process running
2021/12/02 07:44:43 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/02 07:44:45 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:44:55 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/02 07:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:44:57 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:45:07 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/02 07:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (169.14s)
=== RUN   TestN1QLQueryConsistency
2021/12/02 07:45:12 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 07:45:12 App: n1qlquery_consistency got un-deployed
2021/12/02 07:45:12 No eventing-con process running
2021/12/02 07:45:12 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/02 07:45:14 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:45:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:45:20 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:45:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:45:25 Starting bucket ops to source bucket
2021/12/02 07:45:28 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:45:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/02 07:45:35 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 07:45:40 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 07:45:45 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 07:45:50 Waiting for app: n1qlquery_consistency to get deployed
2021/12/02 07:45:50 App: n1qlquery_consistency got deployed
2021/12/02 07:45:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/02 07:46:00 Waiting for dst bucket item count to get to: 5000 curr count: 4093
2021/12/02 07:46:05 Waiting for dst bucket item count to get to: 5000 curr count: 4093
2021/12/02 07:46:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/02 07:46:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/02 07:46:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:46:10 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 07:46:15 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 07:46:20 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 07:46:25 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/02 07:46:25 App: n1qlquery_consistency got un-deployed
2021/12/02 07:46:25 No eventing-con process running
2021/12/02 07:46:25 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/02 07:46:27 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:46:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/02 07:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:46:38 Context: Bucket flush, response code: 200 dump: 
2021/12/02 07:46:48 Waiting for dst bucket item count to get to: 0 curr count: 4236
2021/12/02 07:46:53 Waiting for dst bucket item count to get to: 0 curr count: 1736
2021/12/02 07:46:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.74s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1545.350s

N1ql tests took 0 hours, 25 minutes and 50 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/02 07:48:29 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/02 07:48:46 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 07:48:46 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/02 07:48:46 No eventing-con process running
2021/12/02 07:48:46 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/02 07:48:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:48:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:48:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:48:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:48:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:49:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:49:05 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:49:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 07:49:11 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 07:49:16 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 07:49:21 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 07:49:26 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/02 07:49:26 App: TestDuplicateWithEventingRebIn got deployed
2021/12/02 07:49:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 07:49:26 Starting bucket ops to source bucket
2021/12/02 07:49:40 Removing node(s): [] from the cluster
2021/12/02 07:49:45 Rebalance progress: 25
2021/12/02 07:49:50 Rebalance progress: 25
2021/12/02 07:49:55 Rebalance progress: 25
2021/12/02 07:50:00 Rebalance progress: 25
2021/12/02 07:50:05 Rebalance progress: 75
2021/12/02 07:50:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:50:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:50:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 07:50:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 07:50:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 07:50:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 07:50:16 Rebalance progress: 25
2021/12/02 07:50:21 Rebalance progress: 25
2021/12/02 07:50:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:50:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:50:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 07:50:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 07:50:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 07:50:26 Waiting for src bucket item count to get to: 200000 curr count: 81826
2021/12/02 07:50:31 Waiting for src bucket item count to get to: 200000 curr count: 81826
2021/12/02 07:50:36 Waiting for src bucket item count to get to: 200000 curr count: 97947
2021/12/02 07:50:41 Waiting for src bucket item count to get to: 200000 curr count: 97947
2021/12/02 07:50:46 Waiting for src bucket item count to get to: 200000 curr count: 114254
2021/12/02 07:50:51 Waiting for src bucket item count to get to: 200000 curr count: 114254
2021/12/02 07:50:56 Waiting for src bucket item count to get to: 200000 curr count: 130250
2021/12/02 07:51:01 Waiting for src bucket item count to get to: 200000 curr count: 130250
2021/12/02 07:51:06 Waiting for src bucket item count to get to: 200000 curr count: 146387
2021/12/02 07:51:11 Waiting for src bucket item count to get to: 200000 curr count: 146387
2021/12/02 07:51:16 Waiting for src bucket item count to get to: 200000 curr count: 162479
2021/12/02 07:51:21 Waiting for src bucket item count to get to: 200000 curr count: 162479
2021/12/02 07:51:26 Waiting for src bucket item count to get to: 200000 curr count: 178454
2021/12/02 07:51:31 Waiting for src bucket item count to get to: 200000 curr count: 178454
2021/12/02 07:51:36 Waiting for src bucket item count to get to: 200000 curr count: 194757
2021/12/02 07:51:41 Waiting for src bucket item count to get to: 200000 curr count: 194757
2021/12/02 07:51:51 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:51:56 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:01 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:06 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:11 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:16 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:21 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:26 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:31 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:36 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/02 07:52:36 Post SET ops. Expected item count: 200000 got 200001
2021/12/02 07:52:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:52:41 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 07:52:46 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 07:52:51 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 07:52:56 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/02 07:52:56 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/02 07:52:56 No eventing-con process running
2021/12/02 07:52:56 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/02 07:52:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:53:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:53:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 07:53:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:53:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:53:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:53:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (278.19s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/02 07:53:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 07:53:20 Removing node(s): [] from the cluster
2021/12/02 07:53:25 Rebalance progress: 100
2021/12/02 07:53:30 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 07:53:30 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/02 07:53:30 No eventing-con process running
2021/12/02 07:53:30 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/02 07:53:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:53:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:53:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:53:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:53:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:53:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:53:48 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:53:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 07:53:54 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 07:53:59 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 07:54:04 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 07:54:09 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/02 07:54:09 App: TestDuplicateWithEventingRebOut got deployed
2021/12/02 07:54:09 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 07:54:09 Starting bucket ops to source bucket
2021/12/02 07:54:14 Rebalance progress: 25
2021/12/02 07:54:19 Rebalance progress: 25
2021/12/02 07:54:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:54:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:54:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 07:54:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 07:54:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 07:54:24 Waiting for src bucket item count to get to: 200000 curr count: 15699
2021/12/02 07:54:29 Waiting for src bucket item count to get to: 200000 curr count: 15699
2021/12/02 07:54:34 Waiting for src bucket item count to get to: 200000 curr count: 32085
2021/12/02 07:54:39 Waiting for src bucket item count to get to: 200000 curr count: 32085
2021/12/02 07:54:44 Waiting for src bucket item count to get to: 200000 curr count: 48248
2021/12/02 07:54:49 Waiting for src bucket item count to get to: 200000 curr count: 48248
2021/12/02 07:54:54 Waiting for src bucket item count to get to: 200000 curr count: 48248
2021/12/02 07:54:59 Waiting for src bucket item count to get to: 200000 curr count: 64807
2021/12/02 07:55:04 Waiting for src bucket item count to get to: 200000 curr count: 64807
2021/12/02 07:55:09 Waiting for src bucket item count to get to: 200000 curr count: 80664
2021/12/02 07:55:14 Waiting for src bucket item count to get to: 200000 curr count: 97043
2021/12/02 07:55:19 Waiting for src bucket item count to get to: 200000 curr count: 97043
2021/12/02 07:55:24 Waiting for src bucket item count to get to: 200000 curr count: 113673
2021/12/02 07:55:30 Waiting for src bucket item count to get to: 200000 curr count: 113673
2021/12/02 07:55:35 Waiting for src bucket item count to get to: 200000 curr count: 130311
2021/12/02 07:55:40 Waiting for src bucket item count to get to: 200000 curr count: 130311
2021/12/02 07:55:45 Waiting for src bucket item count to get to: 200000 curr count: 146816
2021/12/02 07:55:50 Waiting for src bucket item count to get to: 200000 curr count: 146816
2021/12/02 07:55:55 Waiting for src bucket item count to get to: 200000 curr count: 162921
2021/12/02 07:56:00 Waiting for src bucket item count to get to: 200000 curr count: 162921
2021/12/02 07:56:05 Waiting for src bucket item count to get to: 200000 curr count: 179072
2021/12/02 07:56:10 Waiting for src bucket item count to get to: 200000 curr count: 179072
2021/12/02 07:56:15 Waiting for src bucket item count to get to: 200000 curr count: 195380
2021/12/02 07:56:20 Waiting for src bucket item count to get to: 200000 curr count: 195380
2021/12/02 07:56:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 07:56:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 07:56:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 07:56:30 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 07:56:35 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 07:56:40 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 07:56:45 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/02 07:56:45 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/02 07:56:45 No eventing-con process running
2021/12/02 07:56:45 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/02 07:56:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:56:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:56:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 07:56:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:56:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:57:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:57:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 07:57:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (233.93s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/02 07:57:18 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 07:57:18 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/02 07:57:18 No eventing-con process running
2021/12/02 07:57:18 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/02 07:57:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:57:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:57:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 07:57:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:57:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:57:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:57:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 07:57:37 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 07:57:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 07:57:43 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 07:57:48 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 07:57:53 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 07:57:58 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/02 07:57:58 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/02 07:57:58 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 07:57:58 Starting bucket ops to source bucket
2021/12/02 07:57:58 Removing node(s): [] from the cluster
2021/12/02 07:58:03 Rebalance progress: 25
2021/12/02 07:58:08 Rebalance progress: 25
2021/12/02 07:58:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:58:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:58:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 07:58:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/02 07:58:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 07:58:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 07:58:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 07:58:25 Rebalance progress: 16.66666666666667
2021/12/02 07:58:30 Rebalance progress: 16.66666666666667
2021/12/02 07:58:35 Rebalance progress: 16.66666666666667
2021/12/02 07:58:40 Rebalance progress: 72.22222222222223
2021/12/02 07:58:45 Rebalance progress: 72.22222222222223
2021/12/02 07:58:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 07:58:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 07:58:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 07:58:55 Rebalance progress: 16.66666666666667
2021/12/02 07:59:00 Rebalance progress: 16.66666666666667
2021/12/02 07:59:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:59:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 07:59:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 07:59:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 07:59:05 Waiting for src bucket item count to get to: 200000 curr count: 90404
2021/12/02 07:59:10 Waiting for src bucket item count to get to: 200000 curr count: 90404
2021/12/02 07:59:15 Waiting for src bucket item count to get to: 200000 curr count: 106394
2021/12/02 07:59:20 Waiting for src bucket item count to get to: 200000 curr count: 106394
2021/12/02 07:59:25 Waiting for src bucket item count to get to: 200000 curr count: 122629
2021/12/02 07:59:30 Waiting for src bucket item count to get to: 200000 curr count: 122629
2021/12/02 07:59:35 Waiting for src bucket item count to get to: 200000 curr count: 138691
2021/12/02 07:59:40 Waiting for src bucket item count to get to: 200000 curr count: 138691
2021/12/02 07:59:45 Waiting for src bucket item count to get to: 200000 curr count: 154651
2021/12/02 07:59:50 Waiting for src bucket item count to get to: 200000 curr count: 154651
2021/12/02 07:59:55 Waiting for src bucket item count to get to: 200000 curr count: 170349
2021/12/02 08:00:01 Waiting for src bucket item count to get to: 200000 curr count: 170349
2021/12/02 08:00:06 Waiting for src bucket item count to get to: 200000 curr count: 186455
2021/12/02 08:00:11 Waiting for src bucket item count to get to: 200000 curr count: 186455
2021/12/02 08:00:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 08:00:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 08:00:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:00:21 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 08:00:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 08:00:31 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 08:00:36 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/02 08:00:36 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/02 08:00:36 No eventing-con process running
2021/12/02 08:00:36 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/02 08:00:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:00:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:00:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:00:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:00:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:00:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:00:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:00:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.98s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/02 08:01:09 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 08:01:09 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/02 08:01:09 No eventing-con process running
2021/12/02 08:01:09 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/02 08:01:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:01:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:01:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:01:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:01:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:01:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:01:28 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:01:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 08:01:34 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 08:01:39 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 08:01:44 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 08:01:49 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/02 08:01:49 App: TestDuplicateWithKVRebIn got deployed
2021/12/02 08:01:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 08:01:49 Starting bucket ops to source bucket
2021/12/02 08:01:49 Removing node(s): [] from the cluster
2021/12/02 08:01:54 Rebalance progress: 16.66666666666667
2021/12/02 08:01:59 Rebalance progress: 16.66666666666667
2021/12/02 08:02:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:02:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:02:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:02:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:02:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:02:09 Rebalance progress: 16.66666666666667
2021/12/02 08:02:14 Rebalance progress: 16.66666666666667
2021/12/02 08:02:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:02:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:02:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:02:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:02:19 Waiting for src bucket item count to get to: 200000 curr count: 30812
2021/12/02 08:02:24 Waiting for src bucket item count to get to: 200000 curr count: 46553
2021/12/02 08:02:29 Waiting for src bucket item count to get to: 200000 curr count: 46553
2021/12/02 08:02:34 Waiting for src bucket item count to get to: 200000 curr count: 62780
2021/12/02 08:02:39 Waiting for src bucket item count to get to: 200000 curr count: 62780
2021/12/02 08:02:44 Waiting for src bucket item count to get to: 200000 curr count: 78163
2021/12/02 08:02:49 Waiting for src bucket item count to get to: 200000 curr count: 78163
2021/12/02 08:02:54 Waiting for src bucket item count to get to: 200000 curr count: 93809
2021/12/02 08:03:00 Waiting for src bucket item count to get to: 200000 curr count: 93809
2021/12/02 08:03:05 Waiting for src bucket item count to get to: 200000 curr count: 93809
2021/12/02 08:03:10 Waiting for src bucket item count to get to: 200000 curr count: 110068
2021/12/02 08:03:15 Waiting for src bucket item count to get to: 200000 curr count: 126395
2021/12/02 08:03:20 Waiting for src bucket item count to get to: 200000 curr count: 126395
2021/12/02 08:03:25 Waiting for src bucket item count to get to: 200000 curr count: 142644
2021/12/02 08:03:30 Waiting for src bucket item count to get to: 200000 curr count: 142644
2021/12/02 08:03:35 Waiting for src bucket item count to get to: 200000 curr count: 159108
2021/12/02 08:03:40 Waiting for src bucket item count to get to: 200000 curr count: 159108
2021/12/02 08:03:45 Waiting for src bucket item count to get to: 200000 curr count: 174646
2021/12/02 08:03:50 Waiting for src bucket item count to get to: 200000 curr count: 174646
2021/12/02 08:03:55 Waiting for src bucket item count to get to: 200000 curr count: 190677
2021/12/02 08:04:00 Waiting for src bucket item count to get to: 200000 curr count: 190677
2021/12/02 08:04:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 08:04:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 08:04:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:04:10 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 08:04:15 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 08:04:20 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 08:04:25 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/02 08:04:25 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/02 08:04:25 No eventing-con process running
2021/12/02 08:04:25 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/02 08:04:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:04:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:04:36 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:04:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:04:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:04:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.05s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/02 08:04:53 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 08:04:54 Removing node(s): [] from the cluster
2021/12/02 08:04:59 Rebalance progress: 100
2021/12/02 08:05:04 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 08:05:04 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/02 08:05:04 No eventing-con process running
2021/12/02 08:05:04 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/02 08:05:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:05:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:05:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:05:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:05:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:05:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:05:22 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:05:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 08:05:28 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 08:05:33 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 08:05:38 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 08:05:43 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/02 08:05:43 App: TestDuplicateWithKVRebOut got deployed
2021/12/02 08:05:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:05:43 Starting bucket ops to source bucket
2021/12/02 08:05:48 Rebalance progress: 16.66666666666667
2021/12/02 08:05:53 Rebalance progress: 16.66666666666667
2021/12/02 08:05:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:05:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:05:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:05:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:05:58 Waiting for src bucket item count to get to: 200000 curr count: 8621
2021/12/02 08:06:03 Waiting for src bucket item count to get to: 200000 curr count: 24501
2021/12/02 08:06:08 Waiting for src bucket item count to get to: 200000 curr count: 24501
2021/12/02 08:06:13 Waiting for src bucket item count to get to: 200000 curr count: 40646
2021/12/02 08:06:18 Waiting for src bucket item count to get to: 200000 curr count: 40646
2021/12/02 08:06:23 Waiting for src bucket item count to get to: 200000 curr count: 56931
2021/12/02 08:06:29 Waiting for src bucket item count to get to: 200000 curr count: 56931
2021/12/02 08:06:34 Waiting for src bucket item count to get to: 200000 curr count: 56931
2021/12/02 08:06:39 Waiting for src bucket item count to get to: 200000 curr count: 72876
2021/12/02 08:06:44 Waiting for src bucket item count to get to: 200000 curr count: 72876
2021/12/02 08:06:49 Waiting for src bucket item count to get to: 200000 curr count: 88898
2021/12/02 08:06:54 Waiting for src bucket item count to get to: 200000 curr count: 88898
2021/12/02 08:06:59 Waiting for src bucket item count to get to: 200000 curr count: 104630
2021/12/02 08:07:04 Waiting for src bucket item count to get to: 200000 curr count: 104630
2021/12/02 08:07:09 Waiting for src bucket item count to get to: 200000 curr count: 120527
2021/12/02 08:07:14 Waiting for src bucket item count to get to: 200000 curr count: 120527
2021/12/02 08:07:19 Waiting for src bucket item count to get to: 200000 curr count: 136289
2021/12/02 08:07:24 Waiting for src bucket item count to get to: 200000 curr count: 136289
2021/12/02 08:07:29 Waiting for src bucket item count to get to: 200000 curr count: 152316
2021/12/02 08:07:34 Waiting for src bucket item count to get to: 200000 curr count: 152316
2021/12/02 08:07:39 Waiting for src bucket item count to get to: 200000 curr count: 168265
2021/12/02 08:07:44 Waiting for src bucket item count to get to: 200000 curr count: 184057
2021/12/02 08:07:49 Waiting for src bucket item count to get to: 200000 curr count: 184057
2021/12/02 08:07:54 Waiting for src bucket item count to get to: 200000 curr count: 199868
2021/12/02 08:07:59 Waiting for src bucket item count to get to: 200000 curr count: 199868
2021/12/02 08:08:04 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 08:08:04 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 08:08:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:08:09 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 08:08:14 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 08:08:19 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 08:08:24 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/02 08:08:24 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/02 08:08:24 No eventing-con process running
2021/12/02 08:08:24 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/02 08:08:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:08:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:08:36 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:08:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:08:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:08:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:08:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.17s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/02 08:08:57 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 08:08:57 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/02 08:08:57 No eventing-con process running
2021/12/02 08:08:57 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/02 08:08:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:09:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:09:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:09:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:09:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:09:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:09:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:09:16 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:09:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 08:09:22 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 08:09:27 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 08:09:32 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 08:09:37 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/02 08:09:37 App: TestDuplicateWithKVSwapReb got deployed
2021/12/02 08:09:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 08:09:37 Starting bucket ops to source bucket
2021/12/02 08:09:37 Removing node(s): [] from the cluster
2021/12/02 08:09:43 Rebalance progress: 16.66666666666667
2021/12/02 08:09:48 Rebalance progress: 16.66666666666667
2021/12/02 08:09:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:09:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:09:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:09:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:09:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 08:09:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:09:58 Rebalance progress: 16.66666666666667
2021/12/02 08:10:03 Rebalance progress: 16.66666666666667
2021/12/02 08:10:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:10:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:10:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:10:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:10:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:10:13 Rebalance progress: 16.66666666666667
2021/12/02 08:10:18 Rebalance progress: 16.66666666666667
2021/12/02 08:10:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:10:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:10:23 Waiting for src bucket item count to get to: 200000 curr count: 64973
2021/12/02 08:10:28 Waiting for src bucket item count to get to: 200000 curr count: 64973
2021/12/02 08:10:33 Waiting for src bucket item count to get to: 200000 curr count: 79950
2021/12/02 08:10:38 Waiting for src bucket item count to get to: 200000 curr count: 79950
2021/12/02 08:10:43 Waiting for src bucket item count to get to: 200000 curr count: 95405
2021/12/02 08:10:48 Waiting for src bucket item count to get to: 200000 curr count: 95405
2021/12/02 08:10:53 Waiting for src bucket item count to get to: 200000 curr count: 111273
2021/12/02 08:10:58 Waiting for src bucket item count to get to: 200000 curr count: 111273
2021/12/02 08:11:03 Waiting for src bucket item count to get to: 200000 curr count: 126967
2021/12/02 08:11:08 Waiting for src bucket item count to get to: 200000 curr count: 126967
2021/12/02 08:11:13 Waiting for src bucket item count to get to: 200000 curr count: 126967
2021/12/02 08:11:18 Waiting for src bucket item count to get to: 200000 curr count: 142770
2021/12/02 08:11:23 Waiting for src bucket item count to get to: 200000 curr count: 142770
2021/12/02 08:11:28 Waiting for src bucket item count to get to: 200000 curr count: 158857
2021/12/02 08:11:33 Waiting for src bucket item count to get to: 200000 curr count: 158857
2021/12/02 08:11:38 Waiting for src bucket item count to get to: 200000 curr count: 174498
2021/12/02 08:11:43 Waiting for src bucket item count to get to: 200000 curr count: 174498
2021/12/02 08:11:48 Waiting for src bucket item count to get to: 200000 curr count: 190459
2021/12/02 08:11:53 Waiting for src bucket item count to get to: 200000 curr count: 190459
2021/12/02 08:11:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 08:11:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 08:11:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:12:03 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 08:12:08 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 08:12:13 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 08:12:18 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/02 08:12:18 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/02 08:12:18 No eventing-con process running
2021/12/02 08:12:18 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/02 08:12:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:12:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:12:30 Waiting for dst bucket item count to get to: 0 curr count: 41960
2021/12/02 08:12:35 Waiting for dst bucket item count to get to: 0 curr count: 41960
2021/12/02 08:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:12:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:12:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/02 08:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.48s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/02 08:12:57 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:12:57 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 08:12:57 No eventing-con process running
2021/12/02 08:12:57 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 08:12:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:13:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:13:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:13:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:13:15 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:13:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 08:13:22 Starting bucket ops to source bucket
2021/12/02 08:13:27 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:13:32 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:13:37 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:13:37 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:13:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:13:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:13:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:13:42 Rebalance progress: 16.66666666666667
2021/12/02 08:13:47 Rebalance progress: 16.66666666666667
2021/12/02 08:13:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:13:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:13:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:13:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:13:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:13:57 Rebalance progress: 16.66666666666667
2021/12/02 08:14:02 Rebalance progress: 16.66666666666667
2021/12/02 08:14:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:14:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:14:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:14:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/02 08:14:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 08:14:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:14:18 Rebalance progress: 12.5
2021/12/02 08:14:23 Rebalance progress: 12.5
2021/12/02 08:14:28 Rebalance progress: 12.5
2021/12/02 08:14:33 Rebalance progress: 70.83333333333334
2021/12/02 08:14:38 Rebalance progress: 70.83333333333334
2021/12/02 08:14:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:14:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:14:48 Rebalance progress: 12.5
2021/12/02 08:14:53 Rebalance progress: 12.5
2021/12/02 08:14:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:14:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:15:03 Rebalance progress: 12.5
2021/12/02 08:15:08 Rebalance progress: 12.5
2021/12/02 08:15:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:15:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:15:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:15:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 08:15:18 Rebalance progress: 12.5
2021/12/02 08:15:23 Rebalance progress: 12.5
2021/12/02 08:15:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:15:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:15:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:15:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:15:33 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:15:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:15:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:15:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:15:48 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 08:15:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:15:48 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 08:15:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:15:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:16:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1469
2021/12/02 08:16:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1469
2021/12/02 08:16:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:16:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:16:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:16:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12633
2021/12/02 08:16:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (209.88s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/02 08:16:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 08:16:27 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/02 08:16:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:16:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/02 08:16:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:16:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:16:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:16:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:16:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:16:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:16:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:16:45 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:16:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/02 08:16:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:16:57 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 08:17:02 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 08:17:07 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/02 08:17:07 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:17:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:17:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:17:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:17:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:17:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 08:17:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:17:13 Rebalance progress: 12.5
2021/12/02 08:17:18 Rebalance progress: 12.5
2021/12/02 08:17:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:17:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 08:17:28 Rebalance progress: 12.5
2021/12/02 08:17:33 Rebalance progress: 12.5
2021/12/02 08:17:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:17:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:17:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 08:17:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 08:17:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 08:17:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/02 08:17:58 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/02 08:17:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:17:58 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/02 08:18:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:18:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:18:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:18:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:18:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:18:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4616
2021/12/02 08:18:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.88s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/02 08:18:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 08:18:27 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 08:18:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:18:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/02 08:18:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:18:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:18:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:18:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:18:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:18:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:18:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:18:45 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:18:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/02 08:18:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:18:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 08:19:01 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 08:19:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/02 08:19:06 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:19:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:19:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:19:12 Rebalance progress: 12.5
2021/12/02 08:19:17 Rebalance progress: 12.5
2021/12/02 08:19:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:19:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:19:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:19:27 Rebalance progress: 12.5
2021/12/02 08:19:32 Rebalance progress: 12.5
2021/12/02 08:19:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:19:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 08:19:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:19:43 Rebalance progress: 12.5
2021/12/02 08:19:48 Rebalance progress: 12.5
2021/12/02 08:19:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:19:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:19:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:19:58 Rebalance progress: 12.5
2021/12/02 08:20:03 Rebalance progress: 12.5
2021/12/02 08:20:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:20:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:20:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:20:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:20:13 Rebalance progress: 12.5
2021/12/02 08:20:18 Rebalance progress: 12.5
2021/12/02 08:20:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:20:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 08:20:28 Rebalance progress: 12.5
2021/12/02 08:20:33 Rebalance progress: 12.5
2021/12/02 08:20:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:20:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:20:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:20:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:20:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 08:20:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 08:20:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 08:20:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/02 08:20:58 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/02 08:20:58 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:20:58 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/02 08:21:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:21:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 565
2021/12/02 08:21:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 565
2021/12/02 08:21:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:21:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:21:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:21:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10638
2021/12/02 08:21:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.08s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/02 08:21:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 08:21:37 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/02 08:21:37 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:21:37 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/02 08:21:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:21:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:21:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:21:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:21:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:21:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:21:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:21:50 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:21:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 08:21:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:22:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 08:22:07 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 08:22:12 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/02 08:22:12 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:22:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:22:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:22:17 Rebalance progress: 12.5
2021/12/02 08:22:22 Rebalance progress: 12.5
2021/12/02 08:22:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:22:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:22:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:22:33 Rebalance progress: 12.5
2021/12/02 08:22:38 Rebalance progress: 12.5
2021/12/02 08:22:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:22:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:22:48 Rebalance progress: 12.5
2021/12/02 08:22:53 Rebalance progress: 12.5
2021/12/02 08:22:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:22:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:22:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:23:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 08:23:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 08:23:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 08:23:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/02 08:23:18 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/02 08:23:18 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:23:18 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/02 08:23:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:23:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:23:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:23:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:23:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:23:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6125
2021/12/02 08:23:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.78s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/02 08:23:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:23:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:23:47 Rebalance progress: 100
2021/12/02 08:23:52 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 08:23:52 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/02 08:23:52 No eventing-con process running
2021/12/02 08:23:52 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/02 08:23:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:23:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:24:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:24:06 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:24:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 08:24:12 Starting bucket ops to source bucket
2021/12/02 08:24:17 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 08:24:22 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 08:24:27 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/02 08:24:27 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:24:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:24:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:24:32 Rebalance progress: 12.5
2021/12/02 08:24:37 Rebalance progress: 12.5
2021/12/02 08:24:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:24:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 08:24:42 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 08:24:47 Rebalance progress: 12.5
2021/12/02 08:24:52 Rebalance progress: 12.5
2021/12/02 08:24:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:24:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:24:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:25:02 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 08:25:08 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 08:25:13 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 08:25:18 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/02 08:25:18 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/02 08:25:18 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:25:18 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/02 08:25:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:25:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:25:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:25:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:25:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:25:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4549
2021/12/02 08:25:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (119.62s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/02 08:25:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:25:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:25:46 Rebalance progress: 100
2021/12/02 08:25:51 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 08:25:51 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/02 08:25:51 No eventing-con process running
2021/12/02 08:25:51 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/02 08:25:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:25:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:26:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:26:05 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:26:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/02 08:26:11 Starting bucket ops to source bucket
2021/12/02 08:26:16 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 08:26:21 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 08:26:26 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/02 08:26:26 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:26:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:26:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:26:32 Rebalance progress: 12.5
2021/12/02 08:26:37 Rebalance progress: 12.5
2021/12/02 08:26:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:26:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 08:26:42 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 08:26:47 Rebalance progress: 12.5
2021/12/02 08:26:52 Rebalance progress: 12.5
2021/12/02 08:26:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:26:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:26:57 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/02 08:27:02 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 08:27:07 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 08:27:12 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 08:27:17 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/02 08:27:17 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/02 08:27:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:27:17 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/02 08:27:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:27:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:27:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37310
2021/12/02 08:27:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:27:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:27:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:27:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 22659
2021/12/02 08:27:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 22659
2021/12/02 08:27:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.38s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/02 08:27:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 08:27:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/02 08:27:56 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:27:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/02 08:27:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:28:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:28:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:28:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:28:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:28:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:28:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:28:14 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:28:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:28:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:28:15 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:28:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 08:28:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:28:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 08:28:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 08:28:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/02 08:28:36 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/02 08:28:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 08:28:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 08:28:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/02 08:28:52 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:28:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:28:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:28:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:28:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:28:57 Rebalance progress: 12.5
2021/12/02 08:29:02 Rebalance progress: 12.5
2021/12/02 08:29:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:29:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:29:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:29:13 Rebalance progress: 12.5
2021/12/02 08:29:18 Rebalance progress: 12.5
2021/12/02 08:29:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:29:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 08:29:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:29:29 Rebalance progress: 12.5
2021/12/02 08:29:34 Rebalance progress: 12.5
2021/12/02 08:29:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:29:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:29:44 Rebalance progress: 12.5
2021/12/02 08:29:49 Rebalance progress: 12.5
2021/12/02 08:29:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:29:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:29:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:29:59 Rebalance progress: 12.5
2021/12/02 08:30:04 Rebalance progress: 12.5
2021/12/02 08:30:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:30:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 08:30:14 Rebalance progress: 12.5
2021/12/02 08:30:19 Rebalance progress: 12.5
2021/12/02 08:30:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:30:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:30:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:30:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:30:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 08:30:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 08:30:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 08:30:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/02 08:30:44 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/02 08:30:44 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:30:45 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/02 08:30:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:30:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:30:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12270
2021/12/02 08:30:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:31:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:31:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:31:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:31:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:31:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 08:31:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 08:31:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 08:31:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/02 08:31:25 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/02 08:31:25 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:31:25 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/02 08:31:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:31:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:31:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:31:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:31:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:31:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.85s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/02 08:31:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 08:31:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/02 08:31:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:31:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/02 08:31:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:31:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:31:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:31:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:32:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:32:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:32:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:32:07 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:32:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:32:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:32:08 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:32:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 08:32:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:32:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 08:32:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 08:32:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/02 08:32:29 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/02 08:32:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 08:32:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 08:32:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/02 08:32:44 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:32:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:32:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:32:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:32:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:32:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/02 08:32:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:32:51 Rebalance progress: 12.5
2021/12/02 08:32:56 Rebalance progress: 12.5
2021/12/02 08:33:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:33:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:33:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:33:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 08:33:06 Rebalance progress: 12.5
2021/12/02 08:33:11 Rebalance progress: 12.5
2021/12/02 08:33:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:33:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:33:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:33:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:33:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 08:33:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 08:33:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 08:33:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/02 08:33:36 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/02 08:33:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:33:36 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/02 08:33:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:33:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:33:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:33:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:33:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:33:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6140
2021/12/02 08:33:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:33:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:34:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 08:34:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 08:34:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 08:34:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/02 08:34:17 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/02 08:34:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:34:18 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/02 08:34:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:34:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:34:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:34:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:34:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:34:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (172.70s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/02 08:34:41 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 08:34:41 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/02 08:34:41 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:34:41 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/02 08:34:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:34:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:34:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:34:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:34:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:34:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:35:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 08:35:00 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/02 08:35:00 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:35:00 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/02 08:35:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:35:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:35:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:35:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:35:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:35:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:35:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:35:13 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:35:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:35:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:35:15 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:35:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 08:35:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:35:26 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 08:35:31 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 08:35:36 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/02 08:35:36 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/02 08:35:41 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 08:35:46 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 08:35:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/02 08:35:51 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:35:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:35:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:35:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:35:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:35:56 Rebalance progress: 12.5
2021/12/02 08:36:01 Rebalance progress: 12.5
2021/12/02 08:36:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:36:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:36:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:36:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:36:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:36:12 Rebalance progress: 12.5
2021/12/02 08:36:17 Rebalance progress: 12.5
2021/12/02 08:36:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:36:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:36:27 Rebalance progress: 12.5
2021/12/02 08:36:32 Rebalance progress: 12.5
2021/12/02 08:36:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:36:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:36:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:36:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:36:42 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 08:36:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 08:36:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 08:36:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/02 08:36:57 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/02 08:36:57 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:36:57 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/02 08:37:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:37:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:37:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 475
2021/12/02 08:37:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 475
2021/12/02 08:37:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:37:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:37:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:37:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7622
2021/12/02 08:37:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:37:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:37:33 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 08:37:38 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 08:37:43 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 08:37:48 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/02 08:37:48 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/02 08:37:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:37:48 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/02 08:37:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:37:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:37:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:37:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:38:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:38:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.25s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 08:38:11 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:38:11 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 08:38:11 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:38:11 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 08:38:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:38:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:38:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:38:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:38:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:38:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:38:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:38:30 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:38:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 08:38:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:38:41 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:38:46 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:38:51 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:38:51 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:38:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:38:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:38:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:38:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:39:12 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:39:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:39:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:39:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:39:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:39:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:39:47 Rebalance progress: 12.5
2021/12/02 08:39:52 Rebalance progress: 12.5
2021/12/02 08:39:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:39:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:39:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:40:03 Rebalance progress: 12.5
2021/12/02 08:40:08 Rebalance progress: 12.5
2021/12/02 08:40:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:40:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:40:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:40:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:40:18 Rebalance progress: 12.5
2021/12/02 08:40:23 Rebalance progress: 12.5
2021/12/02 08:40:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:40:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:40:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:40:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:40:33 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:40:38 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:40:43 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:40:48 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:40:48 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 08:40:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:40:48 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 08:40:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:40:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:41:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 354
2021/12/02 08:41:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 354
2021/12/02 08:41:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:41:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:41:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:41:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11174
2021/12/02 08:41:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.47s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 08:41:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:41:27 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 08:41:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:41:27 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/02 08:41:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:41:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:41:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:41:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:41:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:41:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:41:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:41:45 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:41:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/02 08:41:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:41:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:42:02 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:42:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/02 08:42:07 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:42:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:42:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:42:27 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:42:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:42:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:42:58 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:42:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:43:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:43:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:43:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:43:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:43:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:43:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:43:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:43:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:43:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:43:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:43:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:44:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:44:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:44:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:44:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:44:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:44:47 Rebalance progress: 12.5
2021/12/02 08:44:52 Rebalance progress: 12.5
2021/12/02 08:44:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:44:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:44:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:44:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:45:02 Rebalance progress: 12.5
2021/12/02 08:45:07 Rebalance progress: 12.5
2021/12/02 08:45:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:45:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:45:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:45:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:45:17 Rebalance progress: 12.5
2021/12/02 08:45:22 Rebalance progress: 12.5
2021/12/02 08:45:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:45:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:45:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:45:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:45:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:45:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:45:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:45:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/02 08:45:48 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/02 08:45:48 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:45:48 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/02 08:45:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:45:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:45:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:45:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:46:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:46:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21586
2021/12/02 08:46:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (289.48s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/02 08:46:16 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 08:46:16 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/02 08:46:16 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:46:16 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/02 08:46:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:46:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:46:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:46:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:46:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:46:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:46:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:46:35 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:46:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 08:46:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:46:46 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 08:46:51 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 08:46:56 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/02 08:46:56 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:46:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:46:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:46:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:46:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:47:01 Rebalance progress: 12.5
2021/12/02 08:47:06 Rebalance progress: 12.5
2021/12/02 08:47:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:47:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:47:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:47:26 Rebalance progress: 12.5
2021/12/02 08:47:31 Rebalance progress: 12.5
2021/12/02 08:47:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:47:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 08:47:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:47:42 Rebalance progress: 12.5
2021/12/02 08:47:47 Rebalance progress: 12.5
2021/12/02 08:47:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:47:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:47:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:47:57 Rebalance progress: 12.5
2021/12/02 08:48:02 Rebalance progress: 12.5
2021/12/02 08:48:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:48:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:48:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:48:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:48:12 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 08:48:17 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 08:48:22 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 08:48:27 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/02 08:48:27 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/02 08:48:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:48:27 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/02 08:48:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:48:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:48:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8599
2021/12/02 08:48:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:48:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:48:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:48:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7311
2021/12/02 08:48:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7311
2021/12/02 08:48:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.57s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/02 08:49:06 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 08:49:06 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/02 08:49:06 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:49:06 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/02 08:49:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:49:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:49:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:49:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:49:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:49:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:49:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:49:24 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:49:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/02 08:49:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:49:36 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 08:49:41 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 08:49:46 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/02 08:49:46 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:49:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:49:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:49:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/02 08:49:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:49:51 Rebalance progress: 12.5
2021/12/02 08:49:56 Rebalance progress: 12.5
2021/12/02 08:50:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:50:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/02 08:50:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:50:07 Rebalance progress: 12.5
2021/12/02 08:50:12 Rebalance progress: 12.5
2021/12/02 08:50:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:50:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 08:50:22 Rebalance progress: 12.5
2021/12/02 08:50:27 Rebalance progress: 12.5
2021/12/02 08:50:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:50:32 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:50:37 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 08:50:42 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 08:50:47 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 08:50:52 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/02 08:50:52 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/02 08:50:52 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:50:52 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/02 08:50:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:50:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:51:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6122
2021/12/02 08:51:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6122
2021/12/02 08:51:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:51:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:51:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4239
2021/12/02 08:51:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4239
2021/12/02 08:51:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (149.82s)
=== RUN   TestEventingRebInWithOps
2021/12/02 08:51:36 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 08:51:36 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/02 08:51:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 08:51:36 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/02 08:51:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:51:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:51:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:51:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:51:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:51:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:51:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:51:54 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:51:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 08:52:00 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 08:52:05 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 08:52:10 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 08:52:15 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/02 08:52:16 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/02 08:52:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:52:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 08:52:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 08:52:21 Rebalance progress: 12.5
2021/12/02 08:52:26 Rebalance progress: 12.5
2021/12/02 08:52:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:52:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 08:52:36 Rebalance progress: 12.5
2021/12/02 08:52:41 Rebalance progress: 12.5
2021/12/02 08:52:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 08:52:46 Waiting for src bucket item count to get to: 200000 curr count: 35835
2021/12/02 08:52:51 Waiting for src bucket item count to get to: 200000 curr count: 51526
2021/12/02 08:52:56 Waiting for src bucket item count to get to: 200000 curr count: 51526
2021/12/02 08:53:01 Waiting for src bucket item count to get to: 200000 curr count: 67773
2021/12/02 08:53:06 Waiting for src bucket item count to get to: 200000 curr count: 67773
2021/12/02 08:53:11 Waiting for src bucket item count to get to: 200000 curr count: 83014
2021/12/02 08:53:16 Waiting for src bucket item count to get to: 200000 curr count: 83014
2021/12/02 08:53:21 Waiting for src bucket item count to get to: 200000 curr count: 98939
2021/12/02 08:53:26 Waiting for src bucket item count to get to: 200000 curr count: 98939
2021/12/02 08:53:31 Waiting for src bucket item count to get to: 200000 curr count: 115288
2021/12/02 08:53:36 Waiting for src bucket item count to get to: 200000 curr count: 115288
2021/12/02 08:53:41 Waiting for src bucket item count to get to: 200000 curr count: 131395
2021/12/02 08:53:46 Waiting for src bucket item count to get to: 200000 curr count: 131395
2021/12/02 08:53:51 Waiting for src bucket item count to get to: 200000 curr count: 147414
2021/12/02 08:53:56 Waiting for src bucket item count to get to: 200000 curr count: 147414
2021/12/02 08:54:01 Waiting for src bucket item count to get to: 200000 curr count: 147414
2021/12/02 08:54:06 Waiting for src bucket item count to get to: 200000 curr count: 163351
2021/12/02 08:54:12 Waiting for src bucket item count to get to: 200000 curr count: 163351
2021/12/02 08:54:17 Waiting for src bucket item count to get to: 200000 curr count: 178496
2021/12/02 08:54:22 Waiting for src bucket item count to get to: 200000 curr count: 178496
2021/12/02 08:54:27 Waiting for src bucket item count to get to: 200000 curr count: 194367
2021/12/02 08:54:32 Waiting for src bucket item count to get to: 200000 curr count: 194367
2021/12/02 08:54:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 08:54:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 08:54:37 Starting bucket ops to source bucket
2021/12/02 08:58:28 Waiting for dst bucket item count to get to: 0 curr count: 7476
2021/12/02 08:58:33 Waiting for dst bucket item count to get to: 0 curr count: 7476
2021/12/02 08:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:58:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 08:58:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 08:58:39 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 08:58:44 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 08:58:49 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 08:58:54 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/02 08:58:54 App: TestEventingRebInWithOps got un-deployed
2021/12/02 08:58:54 No eventing-con process running
2021/12/02 08:58:54 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/02 08:58:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:59:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:59:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:59:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:59:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (461.62s)
=== RUN   TestEventingRebOutWithOps
2021/12/02 08:59:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 08:59:13 Removing node(s): [] from the cluster
2021/12/02 08:59:18 Rebalance progress: 100
2021/12/02 08:59:23 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 08:59:23 App: TestEventingRebOutWithOps got un-deployed
2021/12/02 08:59:23 No eventing-con process running
2021/12/02 08:59:23 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/02 08:59:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:59:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:59:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 08:59:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:59:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:59:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 08:59:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 08:59:41 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 08:59:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 08:59:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 08:59:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 08:59:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 09:00:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/02 09:00:02 App: TestEventingRebOutWithOps got deployed
2021/12/02 09:00:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:00:02 Starting bucket ops to source bucket
2021/12/02 09:00:08 Rebalance progress: 12.5
2021/12/02 09:00:13 Rebalance progress: 12.5
2021/12/02 09:00:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:00:18 Waiting for src bucket item count to get to: 200000 curr count: 9897
2021/12/02 09:00:23 Waiting for src bucket item count to get to: 200000 curr count: 25719
2021/12/02 09:00:28 Waiting for src bucket item count to get to: 200000 curr count: 25719
2021/12/02 09:00:33 Waiting for src bucket item count to get to: 200000 curr count: 41808
2021/12/02 09:00:38 Waiting for src bucket item count to get to: 200000 curr count: 41808
2021/12/02 09:00:43 Waiting for src bucket item count to get to: 200000 curr count: 57743
2021/12/02 09:00:48 Waiting for src bucket item count to get to: 200000 curr count: 57743
2021/12/02 09:00:53 Waiting for src bucket item count to get to: 200000 curr count: 57743
2021/12/02 09:00:58 Waiting for src bucket item count to get to: 200000 curr count: 72850
2021/12/02 09:01:03 Waiting for src bucket item count to get to: 200000 curr count: 72850
2021/12/02 09:01:08 Waiting for src bucket item count to get to: 200000 curr count: 88771
2021/12/02 09:01:13 Waiting for src bucket item count to get to: 200000 curr count: 88771
2021/12/02 09:01:18 Waiting for src bucket item count to get to: 200000 curr count: 104675
2021/12/02 09:01:23 Waiting for src bucket item count to get to: 200000 curr count: 104675
2021/12/02 09:01:28 Waiting for src bucket item count to get to: 200000 curr count: 120538
2021/12/02 09:01:33 Waiting for src bucket item count to get to: 200000 curr count: 120538
2021/12/02 09:01:38 Waiting for src bucket item count to get to: 200000 curr count: 136663
2021/12/02 09:01:43 Waiting for src bucket item count to get to: 200000 curr count: 136663
2021/12/02 09:01:48 Waiting for src bucket item count to get to: 200000 curr count: 152871
2021/12/02 09:01:53 Waiting for src bucket item count to get to: 200000 curr count: 152871
2021/12/02 09:01:58 Waiting for src bucket item count to get to: 200000 curr count: 167809
2021/12/02 09:02:03 Waiting for src bucket item count to get to: 200000 curr count: 167809
2021/12/02 09:02:08 Waiting for src bucket item count to get to: 200000 curr count: 183589
2021/12/02 09:02:13 Waiting for src bucket item count to get to: 200000 curr count: 183589
2021/12/02 09:02:18 Waiting for src bucket item count to get to: 200000 curr count: 199407
2021/12/02 09:02:23 Waiting for src bucket item count to get to: 200000 curr count: 199407
2021/12/02 09:02:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 09:02:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 09:02:28 Starting bucket ops to source bucket
2021/12/02 09:06:19 Waiting for dst bucket item count to get to: 0 curr count: 9279
2021/12/02 09:06:24 Waiting for dst bucket item count to get to: 0 curr count: 9279
2021/12/02 09:06:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:06:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:06:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:06:30 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 09:06:35 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 09:06:40 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 09:06:45 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/02 09:06:45 App: TestEventingRebOutWithOps got un-deployed
2021/12/02 09:06:45 No eventing-con process running
2021/12/02 09:06:45 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/02 09:06:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:06:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:06:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:06:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:06:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:06:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (470.84s)
=== RUN   TestEventingSwapRebWithOps
2021/12/02 09:07:08 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 09:07:08 App: TestEventingSwapRebWithOps got un-deployed
2021/12/02 09:07:08 No eventing-con process running
2021/12/02 09:07:08 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/02 09:07:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:07:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:07:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:07:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:07:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:07:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:07:27 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:07:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 09:07:33 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 09:07:38 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 09:07:43 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 09:07:48 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/02 09:07:48 App: TestEventingSwapRebWithOps got deployed
2021/12/02 09:07:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 09:07:48 Starting bucket ops to source bucket
2021/12/02 09:07:48 Removing node(s): [] from the cluster
2021/12/02 09:07:53 Rebalance progress: 12.5
2021/12/02 09:07:58 Rebalance progress: 12.5
2021/12/02 09:08:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:08:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:08:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:08:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 09:08:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:08:09 Rebalance progress: 12.5
2021/12/02 09:08:14 Rebalance progress: 12.5
2021/12/02 09:08:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:08:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:08:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:08:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 09:08:24 Rebalance progress: 12.5
2021/12/02 09:08:29 Rebalance progress: 12.5
2021/12/02 09:08:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:08:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:08:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:08:34 Waiting for src bucket item count to get to: 200000 curr count: 62155
2021/12/02 09:08:39 Waiting for src bucket item count to get to: 200000 curr count: 62155
2021/12/02 09:08:44 Waiting for src bucket item count to get to: 200000 curr count: 77064
2021/12/02 09:08:49 Waiting for src bucket item count to get to: 200000 curr count: 77064
2021/12/02 09:08:54 Waiting for src bucket item count to get to: 200000 curr count: 92837
2021/12/02 09:08:59 Waiting for src bucket item count to get to: 200000 curr count: 92837
2021/12/02 09:09:04 Waiting for src bucket item count to get to: 200000 curr count: 108935
2021/12/02 09:09:09 Waiting for src bucket item count to get to: 200000 curr count: 108935
2021/12/02 09:09:14 Waiting for src bucket item count to get to: 200000 curr count: 125100
2021/12/02 09:09:19 Waiting for src bucket item count to get to: 200000 curr count: 125100
2021/12/02 09:09:24 Waiting for src bucket item count to get to: 200000 curr count: 141263
2021/12/02 09:09:29 Waiting for src bucket item count to get to: 200000 curr count: 141263
2021/12/02 09:09:34 Waiting for src bucket item count to get to: 200000 curr count: 157528
2021/12/02 09:09:39 Waiting for src bucket item count to get to: 200000 curr count: 157528
2021/12/02 09:09:44 Waiting for src bucket item count to get to: 200000 curr count: 172887
2021/12/02 09:09:49 Waiting for src bucket item count to get to: 200000 curr count: 172887
2021/12/02 09:09:54 Waiting for src bucket item count to get to: 200000 curr count: 172887
2021/12/02 09:09:59 Waiting for src bucket item count to get to: 200000 curr count: 188658
2021/12/02 09:10:04 Waiting for src bucket item count to get to: 200000 curr count: 188658
2021/12/02 09:10:09 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 09:10:09 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 09:10:09 Starting bucket ops to source bucket
2021/12/02 09:14:02 Waiting for dst bucket item count to get to: 0 curr count: 13836
2021/12/02 09:14:07 Waiting for dst bucket item count to get to: 0 curr count: 13836
2021/12/02 09:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:14:07 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:14:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:14:12 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 09:14:17 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 09:14:22 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 09:14:27 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/02 09:14:27 App: TestEventingSwapRebWithOps got un-deployed
2021/12/02 09:14:27 No eventing-con process running
2021/12/02 09:14:27 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/02 09:14:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:14:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:14:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:14:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:14:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (462.62s)
=== RUN   TestKVRebInWithOps
2021/12/02 09:14:51 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 09:14:51 App: TestKVRebInWithOps got un-deployed
2021/12/02 09:14:51 No eventing-con process running
2021/12/02 09:14:51 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/02 09:14:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:14:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:14:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:15:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:15:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:15:09 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:15:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 09:15:15 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 09:15:20 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 09:15:25 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 09:15:31 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/02 09:15:31 App: TestKVRebInWithOps got deployed
2021/12/02 09:15:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 09:15:31 Starting bucket ops to source bucket
2021/12/02 09:15:31 Removing node(s): [] from the cluster
2021/12/02 09:15:36 Rebalance progress: 12.5
2021/12/02 09:15:41 Rebalance progress: 12.5
2021/12/02 09:15:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:15:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:15:51 Rebalance progress: 12.5
2021/12/02 09:15:56 Rebalance progress: 12.5
2021/12/02 09:16:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:16:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:16:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:16:01 Waiting for src bucket item count to get to: 200000 curr count: 43459
2021/12/02 09:16:06 Waiting for src bucket item count to get to: 200000 curr count: 43459
2021/12/02 09:16:11 Waiting for src bucket item count to get to: 200000 curr count: 59165
2021/12/02 09:16:16 Waiting for src bucket item count to get to: 200000 curr count: 59165
2021/12/02 09:16:21 Waiting for src bucket item count to get to: 200000 curr count: 74459
2021/12/02 09:16:26 Waiting for src bucket item count to get to: 200000 curr count: 74459
2021/12/02 09:16:31 Waiting for src bucket item count to get to: 200000 curr count: 90245
2021/12/02 09:16:36 Waiting for src bucket item count to get to: 200000 curr count: 90245
2021/12/02 09:16:41 Waiting for src bucket item count to get to: 200000 curr count: 106597
2021/12/02 09:16:46 Waiting for src bucket item count to get to: 200000 curr count: 106597
2021/12/02 09:16:51 Waiting for src bucket item count to get to: 200000 curr count: 122888
2021/12/02 09:16:56 Waiting for src bucket item count to get to: 200000 curr count: 122888
2021/12/02 09:17:01 Waiting for src bucket item count to get to: 200000 curr count: 122888
2021/12/02 09:17:06 Waiting for src bucket item count to get to: 200000 curr count: 139030
2021/12/02 09:17:11 Waiting for src bucket item count to get to: 200000 curr count: 139030
2021/12/02 09:17:16 Waiting for src bucket item count to get to: 200000 curr count: 155271
2021/12/02 09:17:21 Waiting for src bucket item count to get to: 200000 curr count: 155271
2021/12/02 09:17:26 Waiting for src bucket item count to get to: 200000 curr count: 170409
2021/12/02 09:17:31 Waiting for src bucket item count to get to: 200000 curr count: 170409
2021/12/02 09:17:36 Waiting for src bucket item count to get to: 200000 curr count: 186212
2021/12/02 09:17:41 Waiting for src bucket item count to get to: 200000 curr count: 186212
2021/12/02 09:17:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 09:17:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 09:17:46 Starting bucket ops to source bucket
2021/12/02 09:21:38 Waiting for dst bucket item count to get to: 0 curr count: 7458
2021/12/02 09:21:43 Waiting for dst bucket item count to get to: 0 curr count: 7458
2021/12/02 09:21:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:21:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:21:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:21:49 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 09:21:54 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 09:21:59 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 09:22:04 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/02 09:22:04 App: TestKVRebInWithOps got un-deployed
2021/12/02 09:22:04 No eventing-con process running
2021/12/02 09:22:04 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/02 09:22:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:22:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:22:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:22:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:22:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (456.46s)
=== RUN   TestKVRebOutWithOps
2021/12/02 09:22:22 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 09:22:22 Removing node(s): [] from the cluster
2021/12/02 09:22:27 Rebalance progress: 100
2021/12/02 09:22:32 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 09:22:32 App: TestKVRebOutWithOps got un-deployed
2021/12/02 09:22:32 No eventing-con process running
2021/12/02 09:22:32 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/02 09:22:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:22:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:22:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:22:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:22:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:22:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:22:51 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:22:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 09:22:57 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 09:23:02 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 09:23:07 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 09:23:12 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/02 09:23:12 App: TestKVRebOutWithOps got deployed
2021/12/02 09:23:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:23:12 Starting bucket ops to source bucket
2021/12/02 09:23:17 Rebalance progress: 12.5
2021/12/02 09:23:22 Rebalance progress: 12.5
2021/12/02 09:23:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:23:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:23:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:23:27 Waiting for src bucket item count to get to: 200000 curr count: 10229
2021/12/02 09:23:32 Waiting for src bucket item count to get to: 200000 curr count: 26076
2021/12/02 09:23:37 Waiting for src bucket item count to get to: 200000 curr count: 26076
2021/12/02 09:23:42 Waiting for src bucket item count to get to: 200000 curr count: 42234
2021/12/02 09:23:47 Waiting for src bucket item count to get to: 200000 curr count: 42234
2021/12/02 09:23:52 Waiting for src bucket item count to get to: 200000 curr count: 58417
2021/12/02 09:23:57 Waiting for src bucket item count to get to: 200000 curr count: 58417
2021/12/02 09:24:02 Waiting for src bucket item count to get to: 200000 curr count: 58417
2021/12/02 09:24:07 Waiting for src bucket item count to get to: 200000 curr count: 73368
2021/12/02 09:24:12 Waiting for src bucket item count to get to: 200000 curr count: 73368
2021/12/02 09:24:17 Waiting for src bucket item count to get to: 200000 curr count: 89134
2021/12/02 09:24:22 Waiting for src bucket item count to get to: 200000 curr count: 89134
2021/12/02 09:24:27 Waiting for src bucket item count to get to: 200000 curr count: 105221
2021/12/02 09:24:32 Waiting for src bucket item count to get to: 200000 curr count: 105221
2021/12/02 09:24:37 Waiting for src bucket item count to get to: 200000 curr count: 121252
2021/12/02 09:24:42 Waiting for src bucket item count to get to: 200000 curr count: 121252
2021/12/02 09:24:47 Waiting for src bucket item count to get to: 200000 curr count: 137493
2021/12/02 09:24:52 Waiting for src bucket item count to get to: 200000 curr count: 137493
2021/12/02 09:24:57 Waiting for src bucket item count to get to: 200000 curr count: 153505
2021/12/02 09:25:02 Waiting for src bucket item count to get to: 200000 curr count: 153505
2021/12/02 09:25:07 Waiting for src bucket item count to get to: 200000 curr count: 168305
2021/12/02 09:25:12 Waiting for src bucket item count to get to: 200000 curr count: 168305
2021/12/02 09:25:18 Waiting for src bucket item count to get to: 200000 curr count: 184179
2021/12/02 09:25:23 Waiting for src bucket item count to get to: 200000 curr count: 184179
2021/12/02 09:25:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 09:25:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 09:25:28 Starting bucket ops to source bucket
2021/12/02 09:29:20 Waiting for dst bucket item count to get to: 0 curr count: 11008
2021/12/02 09:29:25 Waiting for dst bucket item count to get to: 0 curr count: 11008
2021/12/02 09:29:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:29:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:29:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:29:31 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 09:29:36 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 09:29:41 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 09:29:46 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/02 09:29:46 App: TestKVRebOutWithOps got un-deployed
2021/12/02 09:29:46 No eventing-con process running
2021/12/02 09:29:46 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/02 09:29:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:29:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:29:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:29:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:29:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (462.02s)
=== RUN   TestKVSwapRebWithOps
2021/12/02 09:30:09 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 09:30:09 App: TestKVSwapRebWithOps got un-deployed
2021/12/02 09:30:09 No eventing-con process running
2021/12/02 09:30:09 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/02 09:30:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:30:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:30:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:30:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:30:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:30:28 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:30:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/02 09:30:34 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 09:30:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 09:30:44 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 09:30:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/02 09:30:49 App: TestKVSwapRebWithOps got deployed
2021/12/02 09:30:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 09:30:49 Starting bucket ops to source bucket
2021/12/02 09:30:49 Removing node(s): [] from the cluster
2021/12/02 09:30:55 Rebalance progress: 12.5
2021/12/02 09:31:00 Rebalance progress: 12.5
2021/12/02 09:31:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:31:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:31:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:31:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 09:31:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:31:10 Rebalance progress: 12.5
2021/12/02 09:31:15 Rebalance progress: 12.5
2021/12/02 09:31:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:31:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:31:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:31:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 09:31:25 Rebalance progress: 12.5
2021/12/02 09:31:30 Rebalance progress: 12.5
2021/12/02 09:31:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:31:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:31:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:31:35 Waiting for src bucket item count to get to: 200000 curr count: 60713
2021/12/02 09:31:40 Waiting for src bucket item count to get to: 200000 curr count: 60713
2021/12/02 09:31:45 Waiting for src bucket item count to get to: 200000 curr count: 75441
2021/12/02 09:31:50 Waiting for src bucket item count to get to: 200000 curr count: 75441
2021/12/02 09:31:55 Waiting for src bucket item count to get to: 200000 curr count: 91298
2021/12/02 09:32:00 Waiting for src bucket item count to get to: 200000 curr count: 91298
2021/12/02 09:32:05 Waiting for src bucket item count to get to: 200000 curr count: 107397
2021/12/02 09:32:10 Waiting for src bucket item count to get to: 200000 curr count: 123396
2021/12/02 09:32:15 Waiting for src bucket item count to get to: 200000 curr count: 123396
2021/12/02 09:32:20 Waiting for src bucket item count to get to: 200000 curr count: 139585
2021/12/02 09:32:25 Waiting for src bucket item count to get to: 200000 curr count: 139585
2021/12/02 09:32:30 Waiting for src bucket item count to get to: 200000 curr count: 155661
2021/12/02 09:32:35 Waiting for src bucket item count to get to: 200000 curr count: 155661
2021/12/02 09:32:40 Waiting for src bucket item count to get to: 200000 curr count: 155661
2021/12/02 09:32:45 Waiting for src bucket item count to get to: 200000 curr count: 170611
2021/12/02 09:32:50 Waiting for src bucket item count to get to: 200000 curr count: 170611
2021/12/02 09:32:55 Waiting for src bucket item count to get to: 200000 curr count: 186265
2021/12/02 09:33:00 Waiting for src bucket item count to get to: 200000 curr count: 186265
2021/12/02 09:33:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/02 09:33:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/02 09:33:05 Starting bucket ops to source bucket
2021/12/02 09:36:58 Waiting for dst bucket item count to get to: 0 curr count: 24493
2021/12/02 09:37:03 Waiting for dst bucket item count to get to: 0 curr count: 6619
2021/12/02 09:37:08 Waiting for dst bucket item count to get to: 0 curr count: 6619
2021/12/02 09:37:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:37:08 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:37:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:37:13 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 09:37:18 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 09:37:23 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 09:37:28 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/02 09:37:28 App: TestKVSwapRebWithOps got un-deployed
2021/12/02 09:37:28 No eventing-con process running
2021/12/02 09:37:28 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/02 09:37:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:37:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:37:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:37:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:37:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (462.58s)
=== RUN   TestEventingRebInWithTimers
2021/12/02 09:37:52 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 09:37:52 App: TestEventingRebInWithTimers got un-deployed
2021/12/02 09:37:52 No eventing-con process running
2021/12/02 09:37:52 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/02 09:37:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:37:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:37:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:38:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:38:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:38:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:38:10 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:38:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 09:38:17 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 09:38:22 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 09:38:27 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 09:38:32 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/02 09:38:32 App: TestEventingRebInWithTimers got deployed
2021/12/02 09:38:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 09:38:32 Starting bucket ops to source bucket
2021/12/02 09:38:32 Removing node(s): [] from the cluster
2021/12/02 09:38:37 Rebalance progress: 12.5
2021/12/02 09:38:42 Rebalance progress: 12.5
2021/12/02 09:38:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:38:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:38:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:38:52 Rebalance progress: 12.5
2021/12/02 09:38:57 Rebalance progress: 12.5
2021/12/02 09:39:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:39:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:39:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:39:02 Waiting for src bucket item count to get to: 50000 curr count: 32254
2021/12/02 09:39:07 Waiting for src bucket item count to get to: 50000 curr count: 32254
2021/12/02 09:39:12 Waiting for src bucket item count to get to: 50000 curr count: 42761
2021/12/02 09:39:17 Waiting for src bucket item count to get to: 50000 curr count: 42761
2021/12/02 09:39:27 Waiting for dst bucket item count to get to: 50000 curr count: 39458
2021/12/02 09:39:32 Waiting for dst bucket item count to get to: 50000 curr count: 39458
2021/12/02 09:39:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 09:39:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 09:39:32 Starting bucket ops to source bucket
2021/12/02 09:40:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:40:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:40:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:40:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:41:03 Waiting for dst bucket item count to get to: 0 curr count: 40901
2021/12/02 09:41:08 Waiting for dst bucket item count to get to: 0 curr count: 40901
2021/12/02 09:41:13 Waiting for dst bucket item count to get to: 0 curr count: 23960
2021/12/02 09:41:18 Waiting for dst bucket item count to get to: 0 curr count: 23960
2021/12/02 09:41:23 Waiting for dst bucket item count to get to: 0 curr count: 5872
2021/12/02 09:41:28 Waiting for dst bucket item count to get to: 0 curr count: 5872
2021/12/02 09:41:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:41:28 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:41:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:41:33 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 09:41:38 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 09:41:43 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 09:41:48 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/02 09:41:48 App: TestEventingRebInWithTimers got un-deployed
2021/12/02 09:41:48 No eventing-con process running
2021/12/02 09:41:48 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/02 09:41:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:41:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:41:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:41:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:42:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:42:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (260.13s)
=== RUN   TestEventingRebOutWithTimers
2021/12/02 09:42:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 09:42:07 Removing node(s): [] from the cluster
2021/12/02 09:42:13 Rebalance progress: 100
2021/12/02 09:42:18 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 09:42:18 App: TestEventingRebOutWithTimers got un-deployed
2021/12/02 09:42:18 No eventing-con process running
2021/12/02 09:42:18 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/02 09:42:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:42:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:42:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:42:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:42:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:42:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:42:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:42:36 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:42:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 09:42:42 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 09:42:47 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 09:42:52 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 09:42:57 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/02 09:42:57 App: TestEventingRebOutWithTimers got deployed
2021/12/02 09:42:57 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:42:57 Starting bucket ops to source bucket
2021/12/02 09:43:02 Rebalance progress: 12.5
2021/12/02 09:43:07 Rebalance progress: 12.5
2021/12/02 09:43:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:43:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:43:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:43:12 Waiting for src bucket item count to get to: 50000 curr count: 16479
2021/12/02 09:43:17 Waiting for src bucket item count to get to: 50000 curr count: 16479
2021/12/02 09:43:22 Waiting for src bucket item count to get to: 50000 curr count: 27264
2021/12/02 09:43:27 Waiting for src bucket item count to get to: 50000 curr count: 27264
2021/12/02 09:43:32 Waiting for src bucket item count to get to: 50000 curr count: 37625
2021/12/02 09:43:37 Waiting for src bucket item count to get to: 50000 curr count: 37625
2021/12/02 09:43:42 Waiting for src bucket item count to get to: 50000 curr count: 49606
2021/12/02 09:43:47 Waiting for src bucket item count to get to: 50000 curr count: 49606
2021/12/02 09:43:52 Waiting for src bucket item count to get to: 50000 curr count: 49606
2021/12/02 09:44:02 Waiting for dst bucket item count to get to: 50000 curr count: 48896
2021/12/02 09:44:07 Waiting for dst bucket item count to get to: 50000 curr count: 48896
2021/12/02 09:44:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 09:44:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 09:44:08 Starting bucket ops to source bucket
2021/12/02 09:45:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:45:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:45:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:45:33 Waiting for dst bucket item count to get to: 0 curr count: 45777
2021/12/02 09:45:38 Waiting for dst bucket item count to get to: 0 curr count: 45777
2021/12/02 09:45:43 Waiting for dst bucket item count to get to: 0 curr count: 34081
2021/12/02 09:45:48 Waiting for dst bucket item count to get to: 0 curr count: 34081
2021/12/02 09:45:53 Waiting for dst bucket item count to get to: 0 curr count: 14788
2021/12/02 09:45:58 Waiting for dst bucket item count to get to: 0 curr count: 14788
2021/12/02 09:45:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:45:58 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:45:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:46:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 09:46:09 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 09:46:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 09:46:19 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/02 09:46:19 App: TestEventingRebOutWithTimers got un-deployed
2021/12/02 09:46:19 No eventing-con process running
2021/12/02 09:46:19 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/02 09:46:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:46:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:46:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:46:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:46:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (270.24s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/02 09:46:42 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 09:46:42 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/02 09:46:42 No eventing-con process running
2021/12/02 09:46:42 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/02 09:46:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:46:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:46:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:46:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:47:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:47:01 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:47:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 09:47:07 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 09:47:12 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 09:47:17 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 09:47:22 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/02 09:47:22 App: TestEventingSwapRebWithTimers got deployed
2021/12/02 09:47:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 09:47:22 Starting bucket ops to source bucket
2021/12/02 09:47:23 Removing node(s): [] from the cluster
2021/12/02 09:47:28 Rebalance progress: 12.5
2021/12/02 09:47:33 Rebalance progress: 12.5
2021/12/02 09:47:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:47:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:47:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:47:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 09:47:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:47:43 Rebalance progress: 12.5
2021/12/02 09:47:48 Rebalance progress: 12.5
2021/12/02 09:47:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:47:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:47:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:47:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 09:47:59 Rebalance progress: 12.5
2021/12/02 09:48:04 Rebalance progress: 12.5
2021/12/02 09:48:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:48:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:48:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:48:09 Waiting for src bucket item count to get to: 50000 curr count: 42679
2021/12/02 09:48:19 Waiting for dst bucket item count to get to: 50000 curr count: 34939
2021/12/02 09:48:24 Waiting for dst bucket item count to get to: 50000 curr count: 34939
2021/12/02 09:48:29 Waiting for dst bucket item count to get to: 50000 curr count: 48254
2021/12/02 09:48:34 Waiting for dst bucket item count to get to: 50000 curr count: 48254
2021/12/02 09:48:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 09:48:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 09:48:34 Starting bucket ops to source bucket
2021/12/02 09:49:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:49:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:49:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:50:00 Waiting for dst bucket item count to get to: 0 curr count: 45519
2021/12/02 09:50:05 Waiting for dst bucket item count to get to: 0 curr count: 45519
2021/12/02 09:50:10 Waiting for dst bucket item count to get to: 0 curr count: 32092
2021/12/02 09:50:15 Waiting for dst bucket item count to get to: 0 curr count: 32092
2021/12/02 09:50:20 Waiting for dst bucket item count to get to: 0 curr count: 21533
2021/12/02 09:50:25 Waiting for dst bucket item count to get to: 0 curr count: 21533
2021/12/02 09:50:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:50:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:50:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:50:30 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 09:50:35 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 09:50:40 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 09:50:45 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/02 09:50:45 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/02 09:50:45 No eventing-con process running
2021/12/02 09:50:46 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/02 09:50:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:50:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:50:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:50:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:50:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:50:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (266.91s)
=== RUN   TestKVRebInWithTimers
2021/12/02 09:51:09 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 09:51:09 App: TestKVRebInWithTimers got un-deployed
2021/12/02 09:51:09 No eventing-con process running
2021/12/02 09:51:09 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/02 09:51:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:51:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:51:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:51:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:51:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:51:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:51:28 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:51:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 09:51:34 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 09:51:39 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 09:51:44 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 09:51:49 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/02 09:51:49 App: TestKVRebInWithTimers got deployed
2021/12/02 09:51:49 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 09:51:49 Starting bucket ops to source bucket
2021/12/02 09:51:50 Removing node(s): [] from the cluster
2021/12/02 09:51:55 Rebalance progress: 12.5
2021/12/02 09:52:00 Rebalance progress: 12.5
2021/12/02 09:52:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:52:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:52:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:52:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:52:10 Rebalance progress: 12.5
2021/12/02 09:52:15 Rebalance progress: 12.5
2021/12/02 09:52:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:52:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:52:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:52:20 Waiting for src bucket item count to get to: 50000 curr count: 24678
2021/12/02 09:52:25 Waiting for src bucket item count to get to: 50000 curr count: 34382
2021/12/02 09:52:30 Waiting for src bucket item count to get to: 50000 curr count: 34382
2021/12/02 09:52:35 Waiting for src bucket item count to get to: 50000 curr count: 46575
2021/12/02 09:52:40 Waiting for src bucket item count to get to: 50000 curr count: 46575
2021/12/02 09:52:50 Waiting for dst bucket item count to get to: 50000 curr count: 47663
2021/12/02 09:52:55 Waiting for dst bucket item count to get to: 50000 curr count: 47663
2021/12/02 09:52:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 09:52:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 09:52:55 Starting bucket ops to source bucket
2021/12/02 09:54:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:54:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:54:16 Waiting for dst bucket item count to get to: 0 curr count: 41545
2021/12/02 09:54:21 Waiting for dst bucket item count to get to: 0 curr count: 41545
2021/12/02 09:54:26 Waiting for dst bucket item count to get to: 0 curr count: 41545
2021/12/02 09:54:31 Waiting for dst bucket item count to get to: 0 curr count: 25724
2021/12/02 09:54:36 Waiting for dst bucket item count to get to: 0 curr count: 25724
2021/12/02 09:54:41 Waiting for dst bucket item count to get to: 0 curr count: 13283
2021/12/02 09:54:46 Waiting for dst bucket item count to get to: 0 curr count: 13283
2021/12/02 09:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:54:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:54:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:54:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 09:54:56 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 09:55:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 09:55:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/02 09:55:06 App: TestKVRebInWithTimers got un-deployed
2021/12/02 09:55:06 No eventing-con process running
2021/12/02 09:55:06 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/02 09:55:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:55:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:55:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:55:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:55:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (260.55s)
=== RUN   TestKVRebOutWithTimers
2021/12/02 09:55:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 09:55:25 Removing node(s): [] from the cluster
2021/12/02 09:55:30 Rebalance progress: 100
2021/12/02 09:55:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 09:55:35 App: TestKVRebOutWithTimers got un-deployed
2021/12/02 09:55:35 No eventing-con process running
2021/12/02 09:55:35 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/02 09:55:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:55:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:55:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:55:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:55:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:55:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:55:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:55:54 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 09:55:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 09:56:00 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 09:56:05 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 09:56:10 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 09:56:15 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/02 09:56:15 App: TestKVRebOutWithTimers got deployed
2021/12/02 09:56:15 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 09:56:15 Starting bucket ops to source bucket
2021/12/02 09:56:20 Rebalance progress: 12.5
2021/12/02 09:56:25 Rebalance progress: 12.5
2021/12/02 09:56:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 09:56:30 Waiting for src bucket item count to get to: 50000 curr count: 5420
2021/12/02 09:56:35 Waiting for src bucket item count to get to: 50000 curr count: 19096
2021/12/02 09:56:40 Waiting for src bucket item count to get to: 50000 curr count: 19096
2021/12/02 09:56:45 Waiting for src bucket item count to get to: 50000 curr count: 28711
2021/12/02 09:56:50 Waiting for src bucket item count to get to: 50000 curr count: 41019
2021/12/02 09:56:55 Waiting for src bucket item count to get to: 50000 curr count: 41019
2021/12/02 09:57:05 Waiting for dst bucket item count to get to: 50000 curr count: 40823
2021/12/02 09:57:10 Waiting for dst bucket item count to get to: 50000 curr count: 40823
2021/12/02 09:57:15 Waiting for dst bucket item count to get to: 50000 curr count: 40823
2021/12/02 09:57:20 Waiting for dst bucket item count to get to: 50000 curr count: 48939
2021/12/02 09:57:25 Waiting for dst bucket item count to get to: 50000 curr count: 48939
2021/12/02 09:57:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 09:57:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 09:57:25 Starting bucket ops to source bucket
2021/12/02 09:58:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:58:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:58:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 09:58:51 Waiting for dst bucket item count to get to: 0 curr count: 43070
2021/12/02 09:58:56 Waiting for dst bucket item count to get to: 0 curr count: 43070
2021/12/02 09:59:01 Waiting for dst bucket item count to get to: 0 curr count: 27231
2021/12/02 09:59:06 Waiting for dst bucket item count to get to: 0 curr count: 27231
2021/12/02 09:59:11 Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/02 09:59:16 Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/02 09:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:59:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 09:59:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 09:59:21 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 09:59:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 09:59:31 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 09:59:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/02 09:59:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/02 09:59:36 No eventing-con process running
2021/12/02 09:59:36 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/02 09:59:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:59:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:59:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 09:59:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 09:59:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 09:59:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.82s)
=== RUN   TestKVSwapRebWithTimers
2021/12/02 09:59:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 09:59:59 App: TestKVSwapRebWithTimers got un-deployed
2021/12/02 09:59:59 No eventing-con process running
2021/12/02 09:59:59 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/02 10:00:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:00:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:00:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:00:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:00:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:00:18 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:00:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/02 10:00:24 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 10:00:29 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 10:00:34 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 10:00:39 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/02 10:00:39 App: TestKVSwapRebWithTimers got deployed
2021/12/02 10:00:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:00:39 Starting bucket ops to source bucket
2021/12/02 10:00:39 Removing node(s): [] from the cluster
2021/12/02 10:00:44 Rebalance progress: 12.5
2021/12/02 10:00:49 Rebalance progress: 12.5
2021/12/02 10:00:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:00:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:00:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:00:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:00:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:01:00 Rebalance progress: 12.5
2021/12/02 10:01:05 Rebalance progress: 12.5
2021/12/02 10:01:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:01:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:01:15 Rebalance progress: 12.5
2021/12/02 10:01:20 Rebalance progress: 12.5
2021/12/02 10:01:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:01:25 Waiting for src bucket item count to get to: 50000 curr count: 46847
2021/12/02 10:01:30 Waiting for src bucket item count to get to: 50000 curr count: 46847
2021/12/02 10:01:40 Waiting for dst bucket item count to get to: 50000 curr count: 40406
2021/12/02 10:01:45 Waiting for dst bucket item count to get to: 50000 curr count: 40406
2021/12/02 10:01:50 Waiting for dst bucket item count to get to: 50000 curr count: 48708
2021/12/02 10:01:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:01:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:01:50 Starting bucket ops to source bucket
2021/12/02 10:03:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:03:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:03:11 Waiting for dst bucket item count to get to: 0 curr count: 49355
2021/12/02 10:03:16 Waiting for dst bucket item count to get to: 0 curr count: 49355
2021/12/02 10:03:21 Waiting for dst bucket item count to get to: 0 curr count: 39800
2021/12/02 10:03:26 Waiting for dst bucket item count to get to: 0 curr count: 39800
2021/12/02 10:03:31 Waiting for dst bucket item count to get to: 0 curr count: 29622
2021/12/02 10:03:36 Waiting for dst bucket item count to get to: 0 curr count: 29622
2021/12/02 10:03:41 Waiting for dst bucket item count to get to: 0 curr count: 8120
2021/12/02 10:03:46 Waiting for dst bucket item count to get to: 0 curr count: 8120
2021/12/02 10:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:03:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/02 10:03:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:03:51 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 10:03:56 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 10:04:01 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 10:04:06 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 10:04:11 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 10:04:16 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/02 10:04:16 App: TestKVSwapRebWithTimers got un-deployed
2021/12/02 10:04:16 No eventing-con process running
2021/12/02 10:04:16 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/02 10:04:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:04:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:04:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:04:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:04:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:04:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (280.37s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/02 10:04:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:04:40 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/02 10:04:40 No eventing-con process running
2021/12/02 10:04:40 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/02 10:04:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:04:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:04:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:04:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:04:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:04:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:04:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:04:58 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:05:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 10:05:05 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 10:05:10 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 10:05:15 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 10:05:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/02 10:05:20 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/02 10:05:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 10:05:20 Starting bucket ops to source bucket
2021/12/02 10:05:20 Removing node(s): [] from the cluster
2021/12/02 10:05:25 Rebalance progress: 12.5
2021/12/02 10:05:30 Rebalance progress: 12.5
2021/12/02 10:05:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:05:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:05:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:05:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:05:40 Rebalance progress: 12.5
2021/12/02 10:05:45 Rebalance progress: 12.5
2021/12/02 10:05:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:05:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:05:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:05:50 Waiting for src bucket item count to get to: 50000 curr count: 25071
2021/12/02 10:05:55 Waiting for src bucket item count to get to: 50000 curr count: 37551
2021/12/02 10:06:00 Waiting for src bucket item count to get to: 50000 curr count: 47560
2021/12/02 10:06:05 Waiting for src bucket item count to get to: 50000 curr count: 47560
2021/12/02 10:06:15 Waiting for dst bucket item count to get to: 50000 curr count: 31895
2021/12/02 10:06:20 Waiting for dst bucket item count to get to: 50000 curr count: 31895
2021/12/02 10:06:25 Waiting for dst bucket item count to get to: 50000 curr count: 43044
2021/12/02 10:06:30 Waiting for dst bucket item count to get to: 50000 curr count: 43044
2021/12/02 10:06:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:06:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:06:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:06:36 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:06:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:06:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:06:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:06:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:07:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/02 10:07:01 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/02 10:07:01 No eventing-con process running
2021/12/02 10:07:01 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/02 10:07:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:07:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:07:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:07:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:07:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:07:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:07:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:07:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (174.62s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/02 10:07:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 10:07:30 Removing node(s): [] from the cluster
2021/12/02 10:07:35 Rebalance progress: 100
2021/12/02 10:07:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:07:40 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/02 10:07:40 No eventing-con process running
2021/12/02 10:07:40 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/02 10:07:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:07:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:07:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:07:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:07:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:07:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:07:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:07:58 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:08:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 10:08:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 10:08:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 10:08:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 10:08:20 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/02 10:08:20 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/02 10:08:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:08:20 Starting bucket ops to source bucket
2021/12/02 10:08:25 Rebalance progress: 12.5
2021/12/02 10:08:30 Rebalance progress: 12.5
2021/12/02 10:08:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:08:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:08:35 Waiting for src bucket item count to get to: 50000 curr count: 12326
2021/12/02 10:08:40 Waiting for src bucket item count to get to: 50000 curr count: 12326
2021/12/02 10:08:45 Waiting for src bucket item count to get to: 50000 curr count: 25761
2021/12/02 10:08:50 Waiting for src bucket item count to get to: 50000 curr count: 25761
2021/12/02 10:08:55 Waiting for src bucket item count to get to: 50000 curr count: 37477
2021/12/02 10:09:00 Waiting for src bucket item count to get to: 50000 curr count: 47357
2021/12/02 10:09:05 Waiting for src bucket item count to get to: 50000 curr count: 47357
2021/12/02 10:09:10 Waiting for src bucket item count to get to: 50000 curr count: 47357
2021/12/02 10:09:20 Waiting for dst bucket item count to get to: 50000 curr count: 34942
2021/12/02 10:09:25 Waiting for dst bucket item count to get to: 50000 curr count: 34942
2021/12/02 10:09:30 Waiting for dst bucket item count to get to: 50000 curr count: 42590
2021/12/02 10:09:35 Waiting for dst bucket item count to get to: 50000 curr count: 42590
2021/12/02 10:09:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:09:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:09:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:09:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:09:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:09:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:09:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:10:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:10:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/02 10:10:05 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/02 10:10:05 No eventing-con process running
2021/12/02 10:10:05 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/02 10:10:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:10:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:10:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:10:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:10:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:10:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:10:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.25s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/02 10:10:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:10:39 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/02 10:10:39 No eventing-con process running
2021/12/02 10:10:39 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/02 10:10:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:10:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:10:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:10:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:10:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:10:57 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:10:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 10:11:03 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 10:11:08 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 10:11:13 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 10:11:18 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/02 10:11:19 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/02 10:11:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 10:11:19 Starting bucket ops to source bucket
2021/12/02 10:11:19 Removing node(s): [] from the cluster
2021/12/02 10:11:24 Rebalance progress: 12.5
2021/12/02 10:11:29 Rebalance progress: 12.5
2021/12/02 10:11:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:11:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:11:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:11:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 10:11:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:11:39 Rebalance progress: 12.5
2021/12/02 10:11:44 Rebalance progress: 12.5
2021/12/02 10:11:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:11:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:11:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:11:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:11:55 Rebalance progress: 12.5
2021/12/02 10:12:00 Rebalance progress: 12.5
2021/12/02 10:12:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:12:05 Waiting for src bucket item count to get to: 50000 curr count: 46105
2021/12/02 10:12:10 Waiting for src bucket item count to get to: 50000 curr count: 46105
2021/12/02 10:12:20 Waiting for dst bucket item count to get to: 50000 curr count: 37742
2021/12/02 10:12:25 Waiting for dst bucket item count to get to: 50000 curr count: 37742
2021/12/02 10:12:30 Waiting for dst bucket item count to get to: 50000 curr count: 45022
2021/12/02 10:12:35 Waiting for dst bucket item count to get to: 50000 curr count: 45022
2021/12/02 10:12:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:12:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:12:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:12:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:12:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:12:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:12:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:13:00 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:13:05 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/02 10:13:05 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/02 10:13:05 No eventing-con process running
2021/12/02 10:13:05 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/02 10:13:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:13:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:13:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:13:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:13:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:13:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:13:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.82s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/02 10:13:39 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:13:39 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/02 10:13:39 No eventing-con process running
2021/12/02 10:13:39 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/02 10:13:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:13:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:13:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:13:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:13:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:13:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:13:57 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:13:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 10:14:03 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 10:14:08 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 10:14:13 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 10:14:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/02 10:14:18 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/02 10:14:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:14:18 Starting bucket ops to source bucket
2021/12/02 10:14:19 Removing node(s): [] from the cluster
2021/12/02 10:14:24 Rebalance progress: 12.5
2021/12/02 10:14:29 Rebalance progress: 12.5
2021/12/02 10:14:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:14:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:14:39 Rebalance progress: 12.5
2021/12/02 10:14:44 Rebalance progress: 12.5
2021/12/02 10:14:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:14:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:14:49 Waiting for src bucket item count to get to: 50000 curr count: 26134
2021/12/02 10:14:54 Waiting for src bucket item count to get to: 50000 curr count: 37191
2021/12/02 10:14:59 Waiting for src bucket item count to get to: 50000 curr count: 37191
2021/12/02 10:15:04 Waiting for src bucket item count to get to: 50000 curr count: 48207
2021/12/02 10:15:09 Waiting for src bucket item count to get to: 50000 curr count: 48207
2021/12/02 10:15:19 Waiting for dst bucket item count to get to: 50000 curr count: 34795
2021/12/02 10:15:24 Waiting for dst bucket item count to get to: 50000 curr count: 34795
2021/12/02 10:15:29 Waiting for dst bucket item count to get to: 50000 curr count: 41751
2021/12/02 10:15:34 Waiting for dst bucket item count to get to: 50000 curr count: 41751
2021/12/02 10:15:39 Waiting for dst bucket item count to get to: 50000 curr count: 41751
2021/12/02 10:15:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:15:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:15:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:15:44 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:15:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:15:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:15:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:16:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:16:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/02 10:16:09 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/02 10:16:09 No eventing-con process running
2021/12/02 10:16:09 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/02 10:16:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:16:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:16:21 Waiting for dst bucket item count to get to: 0 curr count: 49514
2021/12/02 10:16:26 Waiting for dst bucket item count to get to: 0 curr count: 49514
2021/12/02 10:16:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:16:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:16:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:16:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:16:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.34s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/02 10:16:43 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:16:43 Removing node(s): [] from the cluster
2021/12/02 10:16:48 Rebalance progress: 100
2021/12/02 10:16:53 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:16:53 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/02 10:16:53 No eventing-con process running
2021/12/02 10:16:53 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/02 10:16:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:17:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:17:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:17:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:17:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:17:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:17:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:17:12 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:17:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 10:17:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 10:17:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 10:17:28 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 10:17:33 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/02 10:17:33 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/02 10:17:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:17:33 Starting bucket ops to source bucket
2021/12/02 10:17:38 Rebalance progress: 12.5
2021/12/02 10:17:43 Rebalance progress: 12.5
2021/12/02 10:17:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:17:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:17:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:17:48 Waiting for src bucket item count to get to: 50000 curr count: 7396
2021/12/02 10:17:53 Waiting for src bucket item count to get to: 50000 curr count: 20919
2021/12/02 10:17:58 Waiting for src bucket item count to get to: 50000 curr count: 20919
2021/12/02 10:18:03 Waiting for src bucket item count to get to: 50000 curr count: 34444
2021/12/02 10:18:08 Waiting for src bucket item count to get to: 50000 curr count: 34444
2021/12/02 10:18:13 Waiting for src bucket item count to get to: 50000 curr count: 44424
2021/12/02 10:18:18 Waiting for src bucket item count to get to: 50000 curr count: 44424
2021/12/02 10:18:23 Waiting for src bucket item count to get to: 50000 curr count: 44424
2021/12/02 10:18:33 Waiting for dst bucket item count to get to: 50000 curr count: 29109
2021/12/02 10:18:38 Waiting for dst bucket item count to get to: 50000 curr count: 29109
2021/12/02 10:18:43 Waiting for dst bucket item count to get to: 50000 curr count: 39828
2021/12/02 10:18:48 Waiting for dst bucket item count to get to: 50000 curr count: 39828
2021/12/02 10:18:53 Waiting for dst bucket item count to get to: 50000 curr count: 49986
2021/12/02 10:18:58 Waiting for dst bucket item count to get to: 50000 curr count: 49986
2021/12/02 10:18:58 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:18:58 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:18:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:19:03 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:19:08 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:19:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:19:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:19:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:19:28 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/02 10:19:28 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/02 10:19:28 No eventing-con process running
2021/12/02 10:19:29 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/02 10:19:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:19:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:19:40 Waiting for dst bucket item count to get to: 0 curr count: 19996
2021/12/02 10:19:45 Waiting for dst bucket item count to get to: 0 curr count: 19996
2021/12/02 10:19:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:19:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:19:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:19:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (199.31s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/02 10:20:07 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:20:07 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/02 10:20:07 No eventing-con process running
2021/12/02 10:20:07 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/02 10:20:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:20:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:20:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:20:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:20:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:20:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:20:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:20:26 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:20:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 10:20:32 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 10:20:37 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 10:20:42 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 10:20:47 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/02 10:20:47 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/02 10:20:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:20:47 Starting bucket ops to source bucket
2021/12/02 10:20:47 Removing node(s): [] from the cluster
2021/12/02 10:20:52 Rebalance progress: 12.5
2021/12/02 10:20:57 Rebalance progress: 12.5
2021/12/02 10:21:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:21:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:21:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:21:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:21:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:21:08 Rebalance progress: 12.5
2021/12/02 10:21:13 Rebalance progress: 12.5
2021/12/02 10:21:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:21:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:21:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:21:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:21:23 Rebalance progress: 12.5
2021/12/02 10:21:28 Rebalance progress: 12.5
2021/12/02 10:21:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:21:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:21:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:21:33 Waiting for src bucket item count to get to: 50000 curr count: 48193
2021/12/02 10:21:38 Waiting for src bucket item count to get to: 50000 curr count: 48193
2021/12/02 10:21:48 Waiting for dst bucket item count to get to: 50000 curr count: 36611
2021/12/02 10:21:53 Waiting for dst bucket item count to get to: 50000 curr count: 36611
2021/12/02 10:21:58 Waiting for dst bucket item count to get to: 50000 curr count: 43657
2021/12/02 10:22:03 Waiting for dst bucket item count to get to: 50000 curr count: 43657
2021/12/02 10:22:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:22:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:22:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:22:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:22:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:22:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:22:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:22:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:22:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/02 10:22:33 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/02 10:22:33 No eventing-con process running
2021/12/02 10:22:33 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/02 10:22:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:22:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:22:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:22:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:22:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:22:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.83s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/02 10:23:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:23:07 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/02 10:23:07 No eventing-con process running
2021/12/02 10:23:07 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/02 10:23:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:23:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:23:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:23:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:23:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:23:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:23:26 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:23:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 10:23:32 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 10:23:37 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 10:23:42 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 10:23:47 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/02 10:23:47 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/02 10:23:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 10:23:47 Starting bucket ops to source bucket
2021/12/02 10:23:47 Removing node(s): [] from the cluster
2021/12/02 10:23:52 Rebalance progress: 12.5
2021/12/02 10:23:57 Rebalance progress: 12.5
2021/12/02 10:24:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:24:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:24:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:24:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:24:08 Rebalance progress: 12.5
2021/12/02 10:24:13 Rebalance progress: 12.5
2021/12/02 10:24:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:24:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:24:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:24:18 Waiting for src bucket item count to get to: 50000 curr count: 25469
2021/12/02 10:24:23 Waiting for src bucket item count to get to: 50000 curr count: 36646
2021/12/02 10:24:28 Waiting for src bucket item count to get to: 50000 curr count: 36646
2021/12/02 10:24:33 Waiting for src bucket item count to get to: 50000 curr count: 47876
2021/12/02 10:24:38 Waiting for src bucket item count to get to: 50000 curr count: 47876
2021/12/02 10:24:48 Waiting for dst bucket item count to get to: 50000 curr count: 47140
2021/12/02 10:24:53 Waiting for dst bucket item count to get to: 50000 curr count: 47140
2021/12/02 10:24:53 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:24:53 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:24:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:24:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:25:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:25:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:25:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:25:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:25:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/02 10:25:23 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/02 10:25:23 No eventing-con process running
2021/12/02 10:25:23 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/02 10:25:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:25:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:25:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:25:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:25:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:25:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:25:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.52s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/02 10:25:52 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 10:25:52 Removing node(s): [] from the cluster
2021/12/02 10:25:57 Rebalance progress: 100
2021/12/02 10:26:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:26:02 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/02 10:26:02 No eventing-con process running
2021/12/02 10:26:02 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/02 10:26:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:26:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:26:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:26:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:26:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:26:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:26:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:26:21 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:26:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 10:26:27 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 10:26:32 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 10:26:37 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 10:26:42 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/02 10:26:42 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/02 10:26:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:26:42 Starting bucket ops to source bucket
2021/12/02 10:26:47 Rebalance progress: 12.5
2021/12/02 10:26:52 Rebalance progress: 12.5
2021/12/02 10:26:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:26:57 Waiting for src bucket item count to get to: 50000 curr count: 9789
2021/12/02 10:27:02 Waiting for src bucket item count to get to: 50000 curr count: 21766
2021/12/02 10:27:07 Waiting for src bucket item count to get to: 50000 curr count: 21766
2021/12/02 10:27:12 Waiting for src bucket item count to get to: 50000 curr count: 33403
2021/12/02 10:27:17 Waiting for src bucket item count to get to: 50000 curr count: 33403
2021/12/02 10:27:22 Waiting for src bucket item count to get to: 50000 curr count: 44033
2021/12/02 10:27:27 Waiting for src bucket item count to get to: 50000 curr count: 44033
2021/12/02 10:27:32 Waiting for src bucket item count to get to: 50000 curr count: 44033
2021/12/02 10:27:42 Waiting for dst bucket item count to get to: 50000 curr count: 41362
2021/12/02 10:27:47 Waiting for dst bucket item count to get to: 50000 curr count: 41362
2021/12/02 10:27:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:27:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:27:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:27:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:27:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:28:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:28:07 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:28:12 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:28:17 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/02 10:28:17 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/02 10:28:17 No eventing-con process running
2021/12/02 10:28:17 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/02 10:28:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:28:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:28:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:28:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:28:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:28:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:28:41 Waiting for dst bucket item count to get to: 0 curr count: 45315
2021/12/02 10:28:46 Waiting for dst bucket item count to get to: 0 curr count: 45315
2021/12/02 10:28:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.39s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/02 10:28:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:28:56 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/02 10:28:56 No eventing-con process running
2021/12/02 10:28:56 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/02 10:28:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:29:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:29:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:29:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:29:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:29:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:29:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:29:15 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:29:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 10:29:21 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 10:29:26 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 10:29:31 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 10:29:36 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/02 10:29:36 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/02 10:29:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/02 10:29:36 Starting bucket ops to source bucket
2021/12/02 10:29:36 Removing node(s): [] from the cluster
2021/12/02 10:29:41 Rebalance progress: 12.5
2021/12/02 10:29:46 Rebalance progress: 12.5
2021/12/02 10:29:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:29:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:29:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:29:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 10:29:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:29:57 Rebalance progress: 12.5
2021/12/02 10:30:02 Rebalance progress: 12.5
2021/12/02 10:30:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:30:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:30:12 Rebalance progress: 12.5
2021/12/02 10:30:17 Rebalance progress: 12.5
2021/12/02 10:30:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:30:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:30:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:30:22 Waiting for src bucket item count to get to: 50000 curr count: 49893
2021/12/02 10:30:27 Waiting for src bucket item count to get to: 50000 curr count: 49893
2021/12/02 10:30:37 Waiting for dst bucket item count to get to: 50000 curr count: 39803
2021/12/02 10:30:42 Waiting for dst bucket item count to get to: 50000 curr count: 39803
2021/12/02 10:30:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:30:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:30:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:30:47 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:30:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:30:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:31:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:31:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:31:12 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/02 10:31:12 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/02 10:31:12 No eventing-con process running
2021/12/02 10:31:12 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/02 10:31:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:31:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:31:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:31:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:31:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:31:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:31:41 Waiting for dst bucket item count to get to: 0 curr count: 35991
2021/12/02 10:31:46 Waiting for dst bucket item count to get to: 0 curr count: 35991
2021/12/02 10:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.87s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/02 10:31:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 10:31:56 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/02 10:31:56 No eventing-con process running
2021/12/02 10:31:56 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/02 10:31:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:32:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:32:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:32:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:32:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:32:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:32:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:32:14 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:32:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/02 10:32:21 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 10:32:26 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 10:32:31 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 10:32:36 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/02 10:32:36 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/02 10:32:36 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:32:36 Starting bucket ops to source bucket
2021/12/02 10:32:36 Removing node(s): [] from the cluster
2021/12/02 10:32:41 Rebalance progress: 12.5
2021/12/02 10:32:46 Rebalance progress: 12.5
2021/12/02 10:32:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:32:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:32:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:32:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:32:56 Rebalance progress: 12.5
2021/12/02 10:33:01 Rebalance progress: 12.5
2021/12/02 10:33:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:33:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:33:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:33:06 Waiting for src bucket item count to get to: 50000 curr count: 28099
2021/12/02 10:33:11 Waiting for src bucket item count to get to: 50000 curr count: 39488
2021/12/02 10:33:16 Waiting for src bucket item count to get to: 50000 curr count: 39488
2021/12/02 10:33:26 Waiting for dst bucket item count to get to: 50000 curr count: 34313
2021/12/02 10:33:31 Waiting for dst bucket item count to get to: 50000 curr count: 34313
2021/12/02 10:33:36 Waiting for dst bucket item count to get to: 50000 curr count: 43776
2021/12/02 10:33:41 Waiting for dst bucket item count to get to: 50000 curr count: 43776
2021/12/02 10:33:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:33:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:33:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:33:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 10:33:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 10:33:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 10:34:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 10:34:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/02 10:34:07 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/02 10:34:07 No eventing-con process running
2021/12/02 10:34:07 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/02 10:34:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:34:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:34:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:34:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:34:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:34:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:34:30 Waiting for dst bucket item count to get to: 0 curr count: 27754
2021/12/02 10:34:36 Waiting for dst bucket item count to get to: 0 curr count: 27754
2021/12/02 10:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.72s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/02 10:34:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:34:41 Removing node(s): [] from the cluster
2021/12/02 10:34:46 Rebalance progress: 100
2021/12/02 10:34:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:34:51 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/02 10:34:51 No eventing-con process running
2021/12/02 10:34:51 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/02 10:34:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:34:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:34:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:35:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:35:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:35:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:35:10 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:35:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/02 10:35:16 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 10:35:21 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 10:35:26 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 10:35:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/02 10:35:31 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/02 10:35:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:35:31 Starting bucket ops to source bucket
2021/12/02 10:35:36 Rebalance progress: 12.5
2021/12/02 10:35:41 Rebalance progress: 12.5
2021/12/02 10:35:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:35:46 Waiting for src bucket item count to get to: 50000 curr count: 11558
2021/12/02 10:35:51 Waiting for src bucket item count to get to: 50000 curr count: 24847
2021/12/02 10:35:56 Waiting for src bucket item count to get to: 50000 curr count: 24847
2021/12/02 10:36:01 Waiting for src bucket item count to get to: 50000 curr count: 34960
2021/12/02 10:36:06 Waiting for src bucket item count to get to: 50000 curr count: 34960
2021/12/02 10:36:11 Waiting for src bucket item count to get to: 50000 curr count: 46435
2021/12/02 10:36:16 Waiting for src bucket item count to get to: 50000 curr count: 46435
2021/12/02 10:36:21 Waiting for src bucket item count to get to: 50000 curr count: 46435
2021/12/02 10:36:31 Waiting for dst bucket item count to get to: 50000 curr count: 37321
2021/12/02 10:36:36 Waiting for dst bucket item count to get to: 50000 curr count: 37321
2021/12/02 10:36:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:36:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:36:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:36:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:36:46 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:36:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:36:56 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:37:01 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:37:06 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/02 10:37:06 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/02 10:37:06 No eventing-con process running
2021/12/02 10:37:06 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/02 10:37:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:37:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:37:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:37:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:37:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:37:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:37:30 Waiting for dst bucket item count to get to: 0 curr count: 6658
2021/12/02 10:37:35 Waiting for dst bucket item count to get to: 0 curr count: 6658
2021/12/02 10:37:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.56s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/02 10:37:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:37:45 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/02 10:37:45 No eventing-con process running
2021/12/02 10:37:45 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/02 10:37:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:37:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:37:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:37:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:37:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:37:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:38:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:38:04 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:38:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/02 10:38:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 10:38:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 10:38:20 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 10:38:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/02 10:38:25 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/02 10:38:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:38:25 Starting bucket ops to source bucket
2021/12/02 10:38:25 Removing node(s): [] from the cluster
2021/12/02 10:38:30 Rebalance progress: 12.5
2021/12/02 10:38:35 Rebalance progress: 12.5
2021/12/02 10:38:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:38:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:38:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:38:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:38:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:38:46 Rebalance progress: 12.5
2021/12/02 10:38:51 Rebalance progress: 12.5
2021/12/02 10:38:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:38:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:39:01 Rebalance progress: 12.5
2021/12/02 10:39:06 Rebalance progress: 12.5
2021/12/02 10:39:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:39:11 Waiting for src bucket item count to get to: 50000 curr count: 49893
2021/12/02 10:39:16 Waiting for src bucket item count to get to: 50000 curr count: 49893
2021/12/02 10:39:26 Waiting for dst bucket item count to get to: 50000 curr count: 43502
2021/12/02 10:39:31 Waiting for dst bucket item count to get to: 50000 curr count: 43502
2021/12/02 10:39:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/02 10:39:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/02 10:39:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:39:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:39:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:39:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:39:51 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:39:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:40:01 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/02 10:40:01 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/02 10:40:01 No eventing-con process running
2021/12/02 10:40:01 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/02 10:40:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:40:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:40:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:40:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/02 10:40:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:40:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:40:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:40:30 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/02 10:40:35 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/02 10:40:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.86s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/02 10:40:45 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:40:45 App: bucket_op_on_update got un-deployed
2021/12/02 10:40:45 No eventing-con process running
2021/12/02 10:40:45 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:40:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:40:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:40:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:40:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:40:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:40:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:41:04 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:41:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 10:41:10 Starting bucket ops to source bucket
2021/12/02 10:41:15 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:41:20 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:41:25 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:41:25 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:41:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:41:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:41:30 Rebalance progress: 12.5
2021/12/02 10:41:35 Rebalance progress: 12.5
2021/12/02 10:41:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:41:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:41:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:41:46 Rebalance progress: 12.5
2021/12/02 10:41:51 Rebalance progress: 12.5
2021/12/02 10:41:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:41:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:41:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 10:41:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:42:01 Rebalance progress: 12.5
2021/12/02 10:42:06 Rebalance progress: 12.5
2021/12/02 10:42:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:42:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:42:16 Rebalance progress: 12.5
2021/12/02 10:42:21 Rebalance progress: 12.5
2021/12/02 10:42:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:42:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:42:31 Rebalance progress: 12.5
2021/12/02 10:42:36 Rebalance progress: 12.5
2021/12/02 10:42:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:42:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 10:42:46 Rebalance progress: 12.5
2021/12/02 10:42:51 Rebalance progress: 12.5
2021/12/02 10:42:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:42:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:42:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:43:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:43:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:43:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:43:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:43:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:43:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:43:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:43:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:43:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 10:43:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:43:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:43:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10670
2021/12/02 10:43:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:43:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:43:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:43:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7282
2021/12/02 10:43:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7282
2021/12/02 10:43:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.72s)
=== RUN   TestKVForceFailover
2021/12/02 10:44:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:44:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:44:11 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:44:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:44:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:44:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:44:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:44:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:44:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:44:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:44:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:44:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:44:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:44:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:44:30 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:44:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 10:44:31 Starting bucket ops to source bucket
2021/12/02 10:44:36 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:44:41 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:44:46 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:44:51 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:44:51 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:44:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/02 10:45:11 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/02 10:45:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/02 10:45:16 Rebalance progress: 12.5
2021/12/02 10:45:21 Rebalance progress: 12.5
2021/12/02 10:45:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:45:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:45:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:45:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/02 10:45:31 Rebalance progress: 12.5
2021/12/02 10:45:36 Rebalance progress: 12.5
2021/12/02 10:45:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:45:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:45:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:45:42 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:45:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:45:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:45:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:46:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:46:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:46:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:46:07 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:46:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 10:46:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:46:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:46:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7062
2021/12/02 10:46:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:46:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:46:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:46:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3303
2021/12/02 10:46:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3303
2021/12/02 10:46:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.75s)
=== RUN   TestKVHardFailoverRecovery
2021/12/02 10:46:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:46:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:46:45 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:46:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:46:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:46:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:46:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:46:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:46:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:47:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:47:04 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:47:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 10:47:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 10:47:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:47:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:47:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:47:25 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:47:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:47:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:47:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:47:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:47:31 Rebalance progress: 12.5
2021/12/02 10:47:36 Rebalance progress: 12.5
2021/12/02 10:47:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:47:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:47:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:47:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/02 10:48:41 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/02 10:48:46 Rebalance progress: 12.5
2021/12/02 10:48:51 Rebalance progress: 12.5
2021/12/02 10:48:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:48:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:48:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:48:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:49:01 Rebalance progress: 12.5
2021/12/02 10:49:06 Rebalance progress: 12.5
2021/12/02 10:49:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:49:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:49:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:49:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:49:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:49:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:49:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:49:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:49:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:49:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:49:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:49:41 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:49:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 10:49:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:49:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:49:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12055
2021/12/02 10:49:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12055
2021/12/02 10:49:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:50:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:50:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:50:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.41s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/02 10:50:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:50:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:50:15 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:50:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:50:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:50:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:50:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:50:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:50:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:50:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:50:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:50:33 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:50:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 10:50:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 10:50:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:50:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:50:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:50:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:50:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:50:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:50:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:50:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:50:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 10:50:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:51:01 Rebalance progress: 12.5
2021/12/02 10:51:06 Rebalance progress: 12.5
2021/12/02 10:51:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:51:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/02 10:51:16 Rebalance progress: 12.5
2021/12/02 10:51:21 Rebalance progress: 12.5
2021/12/02 10:51:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:51:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:51:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:51:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:51:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:51:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:51:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:51:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:51:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:51:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:51:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:51:57 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:51:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 10:51:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:52:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:52:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4675
2021/12/02 10:52:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:52:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:52:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:52:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1762
2021/12/02 10:52:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1762
2021/12/02 10:52:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.49s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/02 10:52:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:52:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:52:35 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:52:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:52:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:52:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:52:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:52:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:52:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:52:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:52:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:52:49 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:52:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 10:52:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 10:53:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:53:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:53:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:53:10 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:53:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:53:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:53:16 Rebalance progress: 12.5
2021/12/02 10:53:21 Rebalance progress: 12.5
2021/12/02 10:53:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:53:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:53:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:53:31 Rebalance progress: 12.5
2021/12/02 10:53:36 Rebalance progress: 12.5
2021/12/02 10:53:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:53:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:53:46 Rebalance progress: 12.5
2021/12/02 10:53:51 Rebalance progress: 12.5
2021/12/02 10:53:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:53:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:53:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:54:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:54:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:54:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:54:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:54:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:54:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:54:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:54:27 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:54:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 10:54:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:54:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:54:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6129
2021/12/02 10:54:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:54:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:54:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:54:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2688
2021/12/02 10:54:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2688
2021/12/02 10:54:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:55:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:55:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:55:00 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:55:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:55:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:55:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.72s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/02 10:55:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:55:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:55:24 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:55:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 10:55:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:55:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:55:43 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:55:43 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 10:55:43 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:55:43 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/02 10:55:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:55:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:55:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:55:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:55:57 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 10:55:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 10:56:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 10:56:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:56:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:56:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 10:56:19 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/02 10:56:24 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 10:56:29 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 10:56:34 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 10:56:34 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:56:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:56:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:56:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 10:56:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:56:39 Rebalance progress: 12.5
2021/12/02 10:56:44 Rebalance progress: 12.5
2021/12/02 10:56:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:56:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:56:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:56:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 10:56:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:56:55 Rebalance progress: 12.5
2021/12/02 10:57:00 Rebalance progress: 12.5
2021/12/02 10:57:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:57:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/02 10:57:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 10:57:10 Rebalance progress: 12.5
2021/12/02 10:57:15 Rebalance progress: 12.5
2021/12/02 10:57:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:57:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 10:57:25 Rebalance progress: 12.5
2021/12/02 10:57:30 Rebalance progress: 12.5
2021/12/02 10:57:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:57:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 10:57:40 Rebalance progress: 12.5
2021/12/02 10:57:45 Rebalance progress: 12.5
2021/12/02 10:57:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:57:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:57:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/02 10:57:56 Rebalance progress: 12.5
2021/12/02 10:58:01 Rebalance progress: 12.5
2021/12/02 10:58:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:58:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:58:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 10:58:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:58:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:58:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:58:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:58:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:58:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:58:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 10:58:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 10:58:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:58:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 10:58:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:58:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:58:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12185
2021/12/02 10:58:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:58:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:58:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:59:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10337
2021/12/02 10:59:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10337
2021/12/02 10:59:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:59:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 10:59:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:59:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:59:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:59:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:59:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:59:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 10:59:36 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 10:59:36 eventingSetup.go:46: No eventing-con process running
2021/12/02 10:59:36 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/02 10:59:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:59:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:59:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 10:59:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 10:59:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 10:59:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.92s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/02 11:00:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:00:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:00:00 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:00:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 11:00:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:00:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:00:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:00:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:00:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:00:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:00:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:00:19 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 11:00:19 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:00:19 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/02 11:00:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:00:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:00:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:00:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:00:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:00:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:00:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:00:32 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:00:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:00:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:00:33 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:00:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/02 11:00:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 11:00:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:00:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:00:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:00:54 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/02 11:00:59 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 11:01:04 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 11:01:10 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/02 11:01:10 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:01:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 11:01:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 11:01:15 Rebalance progress: 12.5
2021/12/02 11:01:20 Rebalance progress: 12.5
2021/12/02 11:01:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:01:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 11:01:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 11:01:30 Rebalance progress: 12.5
2021/12/02 11:01:35 Rebalance progress: 12.5
2021/12/02 11:01:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:01:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 11:01:45 Rebalance progress: 12.5
2021/12/02 11:01:50 Rebalance progress: 12.5
2021/12/02 11:01:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:01:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:02:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:02:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:02:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:02:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:02:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:02:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:02:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:02:26 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:02:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 11:02:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:02:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:02:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:02:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:02:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:02:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7595
2021/12/02 11:02:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:02:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:02:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:02:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:03:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:03:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:03:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:03:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/02 11:03:17 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/02 11:03:17 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:03:17 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/02 11:03:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:03:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:03:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:03:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:03:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:03:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (220.26s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/02 11:03:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:03:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:03:40 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:03:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 11:03:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:03:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:03:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:03:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:03:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:03:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:03:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:03:59 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:04:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 11:04:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 11:04:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:04:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:04:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:04:20 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:04:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:04:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:04:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 11:04:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 11:04:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:04:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:04:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:04:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:04:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:04:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 11:04:56 Rebalance progress: 12.5
2021/12/02 11:05:01 Rebalance progress: 12.5
2021/12/02 11:05:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:05:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:05:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:05:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 11:05:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 11:05:12 Rebalance progress: 12.5
2021/12/02 11:05:17 Rebalance progress: 12.5
2021/12/02 11:05:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:05:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:05:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:05:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 11:05:27 Rebalance progress: 12.5
2021/12/02 11:05:32 Rebalance progress: 12.5
2021/12/02 11:05:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:05:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:05:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:05:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:05:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:05:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:05:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:05:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:06:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:06:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:06:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:06:07 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:06:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 11:06:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:06:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:06:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9161
2021/12/02 11:06:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:06:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:06:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:06:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6327
2021/12/02 11:06:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6327
2021/12/02 11:06:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.45s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/02 11:06:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:06:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:06:46 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:06:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/02 11:06:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:06:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:06:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:06:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:06:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:06:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:07:04 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:07:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 11:07:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/02 11:07:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:07:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:07:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:07:25 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:07:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 11:07:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 11:07:31 Rebalance progress: 12.5
2021/12/02 11:07:36 Rebalance progress: 12.5
2021/12/02 11:07:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:07:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:07:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:07:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 11:07:56 Rebalance progress: 12.5
2021/12/02 11:08:01 Rebalance progress: 12.5
2021/12/02 11:08:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:08:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/02 11:08:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/02 11:08:11 Rebalance progress: 12.5
2021/12/02 11:08:16 Rebalance progress: 12.5
2021/12/02 11:08:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:08:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:08:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:08:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/02 11:08:26 Rebalance progress: 12.5
2021/12/02 11:08:31 Rebalance progress: 12.5
2021/12/02 11:08:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:08:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:08:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:08:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:08:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:08:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:08:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:08:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:09:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:09:02 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:09:02 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:09:02 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 11:09:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:09:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:09:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8580
2021/12/02 11:09:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8580
2021/12/02 11:09:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:09:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:09:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:09:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3687
2021/12/02 11:09:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3687
2021/12/02 11:09:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.73s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/02 11:09:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/02 11:09:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/02 11:09:46 Rebalance progress: 100
2021/12/02 11:09:46 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:10:16 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/02 11:10:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/02 11:10:22 Removing node(s): [] from the cluster
2021/12/02 11:10:22 Starting bucket ops to source bucket
2021/12/02 11:10:27 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/02 11:10:32 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:10:37 Waiting for app: bucket_op_on_update to get deployed
2021/12/02 11:10:37 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:10:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:10:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/02 11:10:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/02 11:10:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:10:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:10:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:10:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:11:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:11:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/02 11:11:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/02 11:11:07 eventingSetup.go:46: No eventing-con process running
2021/12/02 11:11:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/02 11:11:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:11:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:11:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1508
2021/12/02 11:11:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/02 11:11:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/02 11:11:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/02 11:11:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1247
2021/12/02 11:11:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1247
2021/12/02 11:11:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.48s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12185.424s

Rebalance tests took 3 hours, 23 minutes and 7 seconds.

Testing Succeeded


Version: versions-02.12.2021-03.23.cfg
Build Log: make-02.12.2021-03.23.log
Server Log: logs-02.12.2021-03.23.tar.gz

Finished