Building

Started building at 2021/12/09 02:16:58
Calculating base
Updating mirror
Basing run on 7.1.0-1870 f6296f3c7
Updating tree for run 09.12.2021-02.16
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at efc7250, changes since last good build: 
 efc7250f Merge remote-tracking branch 'couchbase/unstable' into HEAD
 4d92f578 MB-43886 : Add bucketInfo to clusterInfoCacheLite
 39e5d769 MB-43886 : Add Collection Info to ClusterInfoCacheLite
 a4eaf13a MB-49780: [System Test] items_count stat negative for multiple indexes
 58027c0d MB-49501 Part 4 (7.1.0 1861): Temp disable CPU throttling for perf tests
kv_engine is at 7eb5198, changes since last good build: 
 7eb5198e6 MB-49802: Pass VB::Commit to doCollectionsMetadataChecks
 55a337c29 MB-47261: ~HashTable to update collection mem_used
 89d2d6911 MB-49839: Remove unused variable
ns_server is at fcadff5, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 2dbd194, changes since last good build: 
 2dbd19461 Slight fix for test case failures in tuq_advise
 b6d233234 check for version 0.0 for cluster run jobs
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: 
 de9f5f4 MB-49931 : Fix "delete from" statement regex
 f37a246 MB-49921 : Return appropriate error for an incorrect N1QL statement
 33225a3 MB-49824 : Fix positional parameters array allocation
 b3770f3 MB-49467 : Manage original and transpiled version of library in store
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable f9b10a85] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable e68b1d02] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 0554d64e] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 796940fc] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 39e67c2d] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable b730708d] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable b781e89f] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable cb1ce49e] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 47a0f10] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/09 02:33:41
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/09 02:35:24 Started functional
2021/12/09 02:35:35 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 02:35:41 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/09 02:35:41 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 02:35:46 Removing node(s): [] from the cluster
2021/12/09 02:35:51 Rebalance progress: 0
2021/12/09 02:35:56 Rebalance progress: 0.5425347222222222
2021/12/09 02:36:01 Rebalance progress: 5.172164351851852
2021/12/09 02:36:06 Rebalance progress: 9.25925925925926
2021/12/09 02:36:11 Rebalance progress: 13.23784722222222
2021/12/09 02:36:16 Rebalance progress: 17.99406828703703
2021/12/09 02:36:21 Rebalance progress: 21.24927662037037
2021/12/09 02:36:26 Rebalance progress: 25.98741319444444
2021/12/09 02:36:31 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/09 02:36:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:36:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:37:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:37:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:37:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:37:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:38:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:38:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:38:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:38:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:39:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:39:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:39:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:39:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:40:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:40:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:40:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:40:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:41:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:41:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:41:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:41:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/09 02:42:07 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:42:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:42:13 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 02:42:18 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 02:42:23 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 02:42:28 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 02:42:28 App: TestAdvancedGetOps got deployed
2021/12/09 02:42:28 Starting bucket ops to source bucket
2021/12/09 02:42:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 02:42:38 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 02:42:43 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 02:42:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 02:42:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:42:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:42:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:42:48 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 02:42:53 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 02:42:58 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 02:43:03 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 02:43:03 App: TestAdvancedGetOps got un-deployed
2021/12/09 02:43:03 No eventing-con process running
2021/12/09 02:43:04 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/09 02:43:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:43:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:43:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:43:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:43:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (410.16s)
=== RUN   TestAdvancedInsertOps
2021/12/09 02:43:27 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:43:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:43:33 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:43:38 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:43:43 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:43:48 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:43:48 App: TestAdvancedInsertOps got deployed
2021/12/09 02:43:48 Starting bucket ops to source bucket
2021/12/09 02:43:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 02:43:59 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 02:43:59 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 02:43:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:43:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:43:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:44:04 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:44:09 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:44:14 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:44:19 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:44:19 App: TestAdvancedInsertOps got un-deployed
2021/12/09 02:44:19 No eventing-con process running
2021/12/09 02:44:19 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/09 02:44:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:44:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 02:44:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 02:44:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:44:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:44:48 Waiting for dst bucket item count to get to: 0 curr count: 584
2021/12/09 02:44:53 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/09 02:44:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:44:53 Testing insert operation on source bucket
2021/12/09 02:44:53 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:44:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:44:59 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:45:04 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:45:09 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:45:14 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 02:45:14 App: TestAdvancedInsertOps got deployed
2021/12/09 02:45:14 Starting bucket ops to source bucket
2021/12/09 02:45:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 02:45:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/09 02:45:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 02:45:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:45:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:45:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:45:30 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:45:35 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:45:40 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:45:45 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 02:45:45 App: TestAdvancedInsertOps got un-deployed
2021/12/09 02:45:45 No eventing-con process running
2021/12/09 02:45:45 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/09 02:45:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:45:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 02:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:45:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.56s)
=== RUN   TestAdvancedUpsertOps
2021/12/09 02:46:04 Starting bucket ops to source bucket
2021/12/09 02:46:04 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:46:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:46:10 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:46:15 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:46:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:46:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:46:25 App: TestAdvancedUpsertOps got deployed
2021/12/09 02:46:25 Starting bucket ops to source bucket
2021/12/09 02:46:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 02:46:36 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 02:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:46:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:46:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:46:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:46:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:46:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:46:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:46:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:46:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/09 02:46:56 No eventing-con process running
2021/12/09 02:46:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/09 02:46:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:47:08 Waiting for dst bucket item count to get to: 0 curr count: 760
2021/12/09 02:47:13 Waiting for dst bucket item count to get to: 0 curr count: 248
2021/12/09 02:47:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:47:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:47:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:47:20 Testing upsert operation on source bucket
2021/12/09 02:47:20 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:47:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:47:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:47:31 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:47:36 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:47:41 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 02:47:41 App: TestAdvancedUpsertOps got deployed
2021/12/09 02:47:41 Starting bucket ops to source bucket
2021/12/09 02:47:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:47:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:47:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:47:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:47:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:47:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:47:57 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:48:02 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 02:48:02 App: TestAdvancedUpsertOps got un-deployed
2021/12/09 02:48:02 No eventing-con process running
2021/12/09 02:48:02 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/09 02:48:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:48:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:48:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:48:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.70s)
=== RUN   TestAdvancedReplaceOps
2021/12/09 02:48:16 Starting bucket ops to source bucket
2021/12/09 02:48:17 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:48:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:48:23 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:48:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:48:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:48:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:48:38 App: TestAdvancedReplaceOps got deployed
2021/12/09 02:48:38 Starting bucket ops to source bucket
2021/12/09 02:48:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 02:48:49 Waiting for dst bucket item count to get to: 0 curr count: 670
2021/12/09 02:48:54 Waiting for dst bucket item count to get to: 0 curr count: 670
2021/12/09 02:48:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:48:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:48:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:48:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:49:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:49:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:49:10 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:49:15 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:49:15 App: TestAdvancedReplaceOps got un-deployed
2021/12/09 02:49:15 No eventing-con process running
2021/12/09 02:49:15 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/09 02:49:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:49:26 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 02:49:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:49:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:49:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:49:33 Testing upsert operation on source bucket
2021/12/09 02:49:33 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:49:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:49:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:49:44 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:49:49 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:49:54 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 02:49:54 App: TestAdvancedReplaceOps got deployed
2021/12/09 02:49:54 Starting bucket ops to source bucket
2021/12/09 02:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:49:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:49:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:49:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:50:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:50:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:50:11 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:50:16 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 02:50:16 App: TestAdvancedReplaceOps got un-deployed
2021/12/09 02:50:16 No eventing-con process running
2021/12/09 02:50:16 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/09 02:50:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:50:27 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/09 02:50:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:50:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:50:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (137.98s)
=== RUN   TestAdvancedDeleteOps
2021/12/09 02:50:34 Starting bucket ops to source bucket
2021/12/09 02:50:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:50:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:50:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:50:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:50:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:50:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:50:55 App: TestAdvancedDeleteOps got deployed
2021/12/09 02:50:55 Starting bucket ops to source bucket
2021/12/09 02:51:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:51:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:51:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:51:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:51:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:51:06 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:51:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:51:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:51:21 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:51:21 App: TestAdvancedDeleteOps got un-deployed
2021/12/09 02:51:21 No eventing-con process running
2021/12/09 02:51:21 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/09 02:51:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:51:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:51:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:51:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:51:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:51:44 Testing delete operation on source bucket
2021/12/09 02:51:44 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:51:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/09 02:51:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:51:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:52:01 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:52:06 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 02:52:06 App: TestAdvancedDeleteOps got deployed
2021/12/09 02:52:06 Starting bucket ops to source bucket
2021/12/09 02:52:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:52:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:52:06 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:52:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:52:11 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:52:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:52:21 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:52:26 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 02:52:26 App: TestAdvancedDeleteOps got un-deployed
2021/12/09 02:52:26 No eventing-con process running
2021/12/09 02:52:26 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/09 02:52:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:52:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:52:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:52:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.53s)
=== RUN   TestEnoentAdvancedGet
2021/12/09 02:52:40 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:52:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/09 02:52:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:52:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:52:56 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:53:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:53:01 App: TestEnoentAdvancedGet got deployed
2021/12/09 02:53:01 Starting bucket ops to source bucket
2021/12/09 02:53:06 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 02:53:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 02:53:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 02:53:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:53:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:53:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:53:16 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:53:21 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:53:26 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:53:31 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:53:31 App: TestEnoentAdvancedGet got un-deployed
2021/12/09 02:53:31 No eventing-con process running
2021/12/09 02:53:31 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/09 02:53:33 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:53:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:53:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:53:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:53:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:54:00 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:54:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:54:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:54:05 Testing get enoent operation on source bucket
2021/12/09 02:54:05 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:54:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/09 02:54:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:54:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:54:21 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:54:26 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 02:54:26 App: TestEnoentAdvancedGet got deployed
2021/12/09 02:54:26 Starting bucket ops to source bucket
2021/12/09 02:54:31 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 02:54:31 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 02:54:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:54:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:54:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:54:36 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:54:41 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:54:46 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:54:51 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 02:54:51 App: TestEnoentAdvancedGet got un-deployed
2021/12/09 02:54:51 No eventing-con process running
2021/12/09 02:54:51 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/09 02:54:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:55:03 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 02:55:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:55:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:55:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.96s)
=== RUN   TestEnoentAdvancedReplace
2021/12/09 02:55:15 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:55:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 02:55:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:55:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:55:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:55:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:55:36 App: TestEnoentAdvancedReplace got deployed
2021/12/09 02:55:36 Starting bucket ops to source bucket
2021/12/09 02:55:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 02:55:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 02:55:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:55:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:55:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:55:46 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:55:51 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:55:56 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:56:01 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:56:01 App: TestEnoentAdvancedReplace got un-deployed
2021/12/09 02:56:01 No eventing-con process running
2021/12/09 02:56:01 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/09 02:56:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:56:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:56:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:56:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:56:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:56:30 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:56:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:56:35 Testing get enoent operation on source bucket
2021/12/09 02:56:35 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:56:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 02:56:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:56:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:56:51 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:56:56 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 02:56:56 App: TestEnoentAdvancedReplace got deployed
2021/12/09 02:56:56 Starting bucket ops to source bucket
2021/12/09 02:57:01 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 02:57:01 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 02:57:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:57:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:57:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:57:06 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:57:11 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:57:16 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:57:21 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 02:57:21 App: TestEnoentAdvancedReplace got un-deployed
2021/12/09 02:57:21 No eventing-con process running
2021/12/09 02:57:21 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/09 02:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:57:33 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 02:57:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:57:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:57:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.01s)
=== RUN   TestEnoentAdvancedDelete
2021/12/09 02:57:45 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:57:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 02:57:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:57:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:58:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:58:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:58:06 App: TestEnoentAdvancedDelete got deployed
2021/12/09 02:58:06 Starting bucket ops to source bucket
2021/12/09 02:58:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 02:58:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 02:58:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:58:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:58:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:58:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:58:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:58:26 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:58:31 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:58:31 App: TestEnoentAdvancedDelete got un-deployed
2021/12/09 02:58:31 No eventing-con process running
2021/12/09 02:58:31 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/09 02:58:33 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:58:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 02:58:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:58:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:58:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 02:59:00 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:59:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 02:59:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 02:59:05 Testing get enoent operation on source bucket
2021/12/09 02:59:05 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 02:59:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 02:59:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:59:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:59:21 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:59:26 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 02:59:26 App: TestEnoentAdvancedDelete got deployed
2021/12/09 02:59:26 Starting bucket ops to source bucket
2021/12/09 02:59:31 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 02:59:31 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 02:59:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 02:59:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 02:59:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 02:59:36 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:59:41 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:59:46 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:59:51 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 02:59:51 App: TestEnoentAdvancedDelete got un-deployed
2021/12/09 02:59:51 No eventing-con process running
2021/12/09 02:59:51 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/09 02:59:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:00:03 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 03:00:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:00:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:00:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:00:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.96s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/09 03:00:15 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:00:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/09 03:00:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:00:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:00:31 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:00:36 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:00:36 App: TestAdvancedInsertKeyExist got deployed
2021/12/09 03:00:36 Starting bucket ops to source bucket
2021/12/09 03:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:00:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:00:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:00:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:00:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:00:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:00:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:00:56 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:00:56 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/09 03:00:56 No eventing-con process running
2021/12/09 03:00:56 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/09 03:00:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:01:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:01:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:01:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:01:10 Pumping document in the destination bucket
2021/12/09 03:01:10 Starting bucket ops to source bucket
2021/12/09 03:01:11 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:01:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/09 03:01:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:01:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:01:27 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:01:32 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 03:01:32 App: TestAdvancedInsertKeyExist got deployed
2021/12/09 03:01:32 Starting bucket ops to source bucket
2021/12/09 03:01:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:01:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:01:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:01:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:01:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:01:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:01:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:01:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:01:57 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:02:02 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 03:02:02 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/09 03:02:02 No eventing-con process running
2021/12/09 03:02:02 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/09 03:02:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:02:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:02:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:02:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:02:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:02:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.78s)
=== RUN   TestExipryGet
2021/12/09 03:02:25 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:02:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/09 03:02:32 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:02:37 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:02:42 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:02:47 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:02:47 App: TestExipryGet got deployed
2021/12/09 03:02:47 Starting bucket ops to source bucket
2021/12/09 03:03:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:03:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:03:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:03:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:03:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:03:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:03:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:03:47 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:03:52 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:03:52 App: TestExipryGet got un-deployed
2021/12/09 03:03:52 No eventing-con process running
2021/12/09 03:03:52 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/09 03:03:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:03:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:04:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:04:06 Pumping document in the destination bucket for expiry get
2021/12/09 03:04:06 Starting bucket ops to source bucket
2021/12/09 03:04:06 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:04:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/09 03:04:12 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:04:17 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:04:22 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:04:27 Waiting for app: TestExipryGet to get deployed
2021/12/09 03:04:28 App: TestExipryGet got deployed
2021/12/09 03:04:28 Starting bucket ops to source bucket
2021/12/09 03:05:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:05:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:05:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:05:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:05:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:05:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:05:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:05:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:05:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:05:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 03:05:43 App: TestExipryGet got un-deployed
2021/12/09 03:05:43 No eventing-con process running
2021/12/09 03:05:43 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/09 03:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:05:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:05:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:06:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (215.90s)
=== RUN   TestExipryInsert
2021/12/09 03:06:01 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:06:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/09 03:06:08 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:06:13 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:06:18 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:06:23 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:06:23 App: TestExipryInsert got deployed
2021/12/09 03:06:23 Starting bucket ops to source bucket
2021/12/09 03:07:08 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 03:07:13 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 03:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:07:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:07:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:07:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:07:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:07:23 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:07:28 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:07:33 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:07:33 App: TestExipryInsert got un-deployed
2021/12/09 03:07:33 No eventing-con process running
2021/12/09 03:07:33 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/09 03:07:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:07:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:07:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:07:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:07:47 Testing on the destination bucket for expiry insert
2021/12/09 03:07:47 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:07:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/09 03:07:53 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:07:58 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:08:03 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:08:08 Waiting for app: TestExipryInsert to get deployed
2021/12/09 03:08:08 App: TestExipryInsert got deployed
2021/12/09 03:08:08 Starting bucket ops to source bucket
2021/12/09 03:08:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:08:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:08:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:08:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:08:58 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:08:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:09:03 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:09:08 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:09:13 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:09:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 03:09:18 App: TestExipryInsert got un-deployed
2021/12/09 03:09:18 No eventing-con process running
2021/12/09 03:09:18 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/09 03:09:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:09:30 Waiting for dst bucket item count to get to: 0 curr count: 64
2021/12/09 03:09:35 Waiting for dst bucket item count to get to: 0 curr count: 64
2021/12/09 03:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:09:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:09:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.43s)
=== RUN   TestExipryReplace
2021/12/09 03:09:42 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:09:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 03:09:48 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:09:53 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:09:58 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:10:03 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:10:03 App: TestExipryReplace got deployed
2021/12/09 03:10:03 Starting bucket ops to source bucket
2021/12/09 03:10:48 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:10:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:10:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:10:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:10:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:10:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:10:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:11:03 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:11:08 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:11:13 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:11:13 App: TestExipryReplace got un-deployed
2021/12/09 03:11:13 No eventing-con process running
2021/12/09 03:11:13 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/09 03:11:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:11:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:11:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:11:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:11:27 Testing on the destination bucket for expiry insert
2021/12/09 03:11:27 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:11:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 03:11:33 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:11:38 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:11:43 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:11:48 Waiting for app: TestExipryReplace to get deployed
2021/12/09 03:11:48 App: TestExipryReplace got deployed
2021/12/09 03:11:48 Starting bucket ops to source bucket
2021/12/09 03:12:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:12:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:12:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:12:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:12:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:12:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:12:44 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:12:49 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:12:54 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:12:59 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 03:12:59 App: TestExipryReplace got un-deployed
2021/12/09 03:12:59 No eventing-con process running
2021/12/09 03:12:59 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/09 03:13:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:13:11 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/09 03:13:16 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/09 03:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:13:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.50s)
=== RUN   TestExipryUpsert
2021/12/09 03:13:22 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:13:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/09 03:13:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:13:34 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:13:39 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:13:44 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:13:44 App: TestExipryUpsert got deployed
2021/12/09 03:13:44 Starting bucket ops to source bucket
2021/12/09 03:14:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:14:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:14:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:14:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:14:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:14:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:14:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:14:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:14:49 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:14:54 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:14:54 App: TestExipryUpsert got un-deployed
2021/12/09 03:14:54 No eventing-con process running
2021/12/09 03:14:54 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/09 03:14:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:15:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:15:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:15:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:15:08 Testing on the destination bucket for expiry insert
2021/12/09 03:15:08 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:15:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/09 03:15:14 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:15:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:15:24 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:15:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 03:15:29 App: TestExipryUpsert got deployed
2021/12/09 03:15:29 Starting bucket ops to source bucket
2021/12/09 03:16:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:16:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:16:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:16:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:16:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:16:24 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:16:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:16:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:16:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 03:16:39 App: TestExipryUpsert got un-deployed
2021/12/09 03:16:39 No eventing-con process running
2021/12/09 03:16:39 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/09 03:16:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:16:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:16:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:16:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:16:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:17:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.50s)
=== RUN   TestCasReplace
2021/12/09 03:17:03 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:17:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 03:17:09 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:17:14 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:17:19 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:17:24 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:17:24 App: TestCasReplace got deployed
2021/12/09 03:17:24 Starting bucket ops to source bucket
2021/12/09 03:17:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:17:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:17:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:17:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:17:29 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:17:34 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:17:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:17:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:17:44 App: TestCasReplace got un-deployed
2021/12/09 03:17:44 No eventing-con process running
2021/12/09 03:17:44 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/09 03:17:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:17:56 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/09 03:17:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:17:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:18:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:18:03 Testing on the destination bucket for upsert with cas
2021/12/09 03:18:03 Starting bucket ops to source bucket
2021/12/09 03:18:03 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:18:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 03:18:09 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:18:14 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:18:19 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:18:24 Waiting for app: TestCasReplace to get deployed
2021/12/09 03:18:24 App: TestCasReplace got deployed
2021/12/09 03:18:24 Starting bucket ops to source bucket
2021/12/09 03:18:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:18:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:18:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:18:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:18:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:18:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:18:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:18:50 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:18:55 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 03:18:55 App: TestCasReplace got un-deployed
2021/12/09 03:18:55 No eventing-con process running
2021/12/09 03:18:55 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/09 03:18:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:19:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:19:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:19:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:19:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.24s)
=== RUN   TestCasDelete
2021/12/09 03:19:13 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:19:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 03:19:19 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:19:24 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:19:29 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:19:34 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:19:34 App: TestCasDelete got deployed
2021/12/09 03:19:34 Starting bucket ops to source bucket
2021/12/09 03:19:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:19:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:19:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:19:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:19:40 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:19:45 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:19:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:19:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:19:55 App: TestCasDelete got un-deployed
2021/12/09 03:19:55 No eventing-con process running
2021/12/09 03:19:55 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/09 03:19:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:20:06 Waiting for dst bucket item count to get to: 0 curr count: 7
2021/12/09 03:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:20:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:20:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:20:13 Testing on the destination bucket for delete with cas
2021/12/09 03:20:13 Starting bucket ops to source bucket
2021/12/09 03:20:13 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:20:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 03:20:19 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:20:24 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:20:29 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:20:34 Waiting for app: TestCasDelete to get deployed
2021/12/09 03:20:34 App: TestCasDelete got deployed
2021/12/09 03:20:34 Starting bucket ops to source bucket
2021/12/09 03:20:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:20:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:20:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:20:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 03:20:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:20:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:20:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:21:00 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:21:05 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 03:21:05 App: TestCasDelete got un-deployed
2021/12/09 03:21:05 No eventing-con process running
2021/12/09 03:21:05 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/09 03:21:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:21:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.15s)
=== RUN   TestCountersIncrement
2021/12/09 03:21:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 03:21:36 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 03:21:36 Removing node(s): [] from the cluster
2021/12/09 03:21:41 Rebalance progress: 100
2021/12/09 03:21:41 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:21:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/09 03:21:47 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:21:52 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:21:57 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:22:02 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:22:03 App: TestCountersIncrement got deployed
2021/12/09 03:22:03 Starting bucket ops to source bucket
2021/12/09 03:22:09 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 03:22:14 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 03:22:19 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 03:22:19 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 03:22:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:22:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:22:24 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:22:29 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:22:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:22:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:22:39 App: TestCountersIncrement got un-deployed
2021/12/09 03:22:39 No eventing-con process running
2021/12/09 03:22:39 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/09 03:22:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:22:51 Waiting for dst bucket item count to get to: 0 curr count: 881
2021/12/09 03:22:56 Waiting for dst bucket item count to get to: 0 curr count: 881
2021/12/09 03:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:22:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:23:07 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 03:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:23:07 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:23:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/09 03:23:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:23:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:23:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:23:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 03:23:29 App: TestCountersIncrement got deployed
2021/12/09 03:23:29 Starting bucket ops to source bucket
2021/12/09 03:23:35 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 03:23:40 Waiting for dst bucket item count to get to: 2048 curr count: 528
2021/12/09 03:23:45 Waiting for dst bucket item count to get to: 2048 curr count: 1552
2021/12/09 03:23:45 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 03:23:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:23:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:23:50 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:23:55 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:24:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:24:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 03:24:05 App: TestCountersIncrement got un-deployed
2021/12/09 03:24:05 No eventing-con process running
2021/12/09 03:24:05 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/09 03:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:24:17 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 03:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:24:29 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 03:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:24:29 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/09 03:24:34 Rebalance progress: 100
--- PASS: TestCountersIncrement (190.43s)
=== RUN   TestCountersDecrement
2021/12/09 03:24:34 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 03:24:34 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 03:24:35 Removing node(s): [] from the cluster
2021/12/09 03:24:40 Rebalance progress: 100
2021/12/09 03:24:40 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:24:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/09 03:24:46 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:24:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:24:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:25:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:25:01 App: TestCountersDecrement got deployed
2021/12/09 03:25:01 Starting bucket ops to source bucket
2021/12/09 03:25:07 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 03:25:12 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 03:25:17 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 03:25:17 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 03:25:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:25:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:25:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:25:27 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:25:32 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:25:37 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:25:37 App: TestCountersDecrement got un-deployed
2021/12/09 03:25:37 No eventing-con process running
2021/12/09 03:25:37 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/09 03:25:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:25:49 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/09 03:25:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 03:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:25:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:26:06 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 03:26:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:26:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:26:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/09 03:26:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:26:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:26:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:26:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 03:26:27 App: TestCountersDecrement got deployed
2021/12/09 03:26:27 Starting bucket ops to source bucket
2021/12/09 03:26:33 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 03:26:38 Waiting for dst bucket item count to get to: 2048 curr count: 1040
2021/12/09 03:26:43 Waiting for dst bucket item count to get to: 2048 curr count: 1040
2021/12/09 03:26:43 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 03:26:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:26:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:26:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:26:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:26:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:27:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 03:27:03 App: TestCountersDecrement got un-deployed
2021/12/09 03:27:03 No eventing-con process running
2021/12/09 03:27:04 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/09 03:27:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:27:15 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 03:27:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:27:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:27:27 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 03:27:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:27:27 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/09 03:27:32 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.46s)
=== RUN   TestCacheGet
2021/12/09 03:27:32 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:27:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/09 03:27:38 Waiting for app: TestCacheGet to get deployed
2021/12/09 03:27:43 Waiting for app: TestCacheGet to get deployed
2021/12/09 03:27:48 Waiting for app: TestCacheGet to get deployed
2021/12/09 03:27:53 Waiting for app: TestCacheGet to get deployed
2021/12/09 03:27:53 App: TestCacheGet got deployed
2021/12/09 03:27:53 Starting bucket ops to source bucket
2021/12/09 03:27:58 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 03:28:03 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 03:28:08 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 03:28:13 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/09 03:28:18 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/09 03:28:23 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/09 03:28:23 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/09 03:28:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:28:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:28:29 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 03:28:34 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 03:28:39 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 03:28:44 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 03:28:44 App: TestCacheGet got un-deployed
2021/12/09 03:28:44 No eventing-con process running
2021/12/09 03:28:44 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/09 03:28:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:28:55 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 03:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:28:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:29:07 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 03:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.15s)
=== RUN   TestCacheRYOW
2021/12/09 03:29:07 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:29:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/09 03:29:13 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 03:29:18 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 03:29:23 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 03:29:28 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 03:29:28 App: TestCacheRYOW got deployed
2021/12/09 03:29:28 Starting bucket ops to source bucket
2021/12/09 03:29:34 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/09 03:29:39 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/09 03:29:39 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/09 03:29:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:29:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:29:44 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 03:29:49 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 03:29:54 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 03:29:59 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 03:29:59 App: TestCacheRYOW got un-deployed
2021/12/09 03:29:59 No eventing-con process running
2021/12/09 03:29:59 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/09 03:30:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:30:11 Waiting for dst bucket item count to get to: 0 curr count: 42
2021/12/09 03:30:16 Waiting for dst bucket item count to get to: 0 curr count: 42
2021/12/09 03:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:30:27 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/09 03:30:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.14s)
=== RUN   TestCacheExpiry
2021/12/09 03:30:27 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:30:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/09 03:30:34 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 03:30:39 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 03:30:44 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 03:30:49 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 03:30:49 App: TestCacheExpiry got deployed
2021/12/09 03:30:49 Starting bucket ops to source bucket
2021/12/09 03:30:54 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:30:59 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/09 03:31:04 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/09 03:31:09 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/09 03:31:14 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/09 03:31:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:31:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:31:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:31:19 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 03:31:24 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 03:31:29 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 03:31:34 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 03:31:34 App: TestCacheExpiry got un-deployed
2021/12/09 03:31:34 No eventing-con process running
2021/12/09 03:31:34 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/09 03:31:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:31:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:31:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:31:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.16s)
=== RUN   TestCacheExpiryCustom
2021/12/09 03:31:58 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:31:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/09 03:32:04 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 03:32:09 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 03:32:14 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 03:32:19 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 03:32:19 App: TestCacheExpiryCustom got deployed
2021/12/09 03:32:19 Starting bucket ops to source bucket
2021/12/09 03:32:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:32:29 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/09 03:32:34 Waiting for dst bucket item count to get to: 100 curr count: 41
2021/12/09 03:32:39 Waiting for dst bucket item count to get to: 100 curr count: 85
2021/12/09 03:32:44 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/09 03:32:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:32:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:32:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:32:49 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 03:32:54 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 03:32:59 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 03:33:04 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 03:33:04 App: TestCacheExpiryCustom got un-deployed
2021/12/09 03:33:04 No eventing-con process running
2021/12/09 03:33:04 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/09 03:33:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:33:16 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:33:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:33:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.10s)
=== RUN   TestCacheOverflowCustom
2021/12/09 03:33:28 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:33:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/09 03:33:34 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 03:33:39 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 03:33:44 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 03:33:49 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 03:33:49 App: TestCacheOverflowCustom got deployed
2021/12/09 03:33:49 Starting bucket ops to source bucket
2021/12/09 03:33:54 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/09 03:33:59 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/09 03:34:04 Waiting for dst bucket item count to get to: 35 curr count: 9
2021/12/09 03:34:09 Waiting for dst bucket item count to get to: 35 curr count: 21
2021/12/09 03:34:14 Waiting for dst bucket item count to get to: 35 curr count: 29
2021/12/09 03:34:19 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/09 03:34:24 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/09 03:34:24 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/09 03:34:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:34:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:34:29 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 03:34:34 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 03:34:39 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 03:34:44 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 03:34:44 App: TestCacheOverflowCustom got un-deployed
2021/12/09 03:34:44 No eventing-con process running
2021/12/09 03:34:44 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/09 03:34:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:34:56 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/09 03:34:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:34:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:35:08 Waiting for dst bucket item count to get to: 0 curr count: 23
2021/12/09 03:35:13 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/09 03:35:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (105.22s)
=== RUN   TestStrictMode
2021/12/09 03:35:13 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:35:14 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.05s)
=== RUN   TestDataTypes
2021/12/09 03:35:14 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:35:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/09 03:35:20 Waiting for app: TestDataTypes to get deployed
2021/12/09 03:35:25 Waiting for app: TestDataTypes to get deployed
2021/12/09 03:35:30 Waiting for app: TestDataTypes to get deployed
2021/12/09 03:35:35 Waiting for app: TestDataTypes to get deployed
2021/12/09 03:35:35 App: TestDataTypes got deployed
2021/12/09 03:35:35 Starting bucket ops to source bucket
2021/12/09 03:35:40 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/09 03:35:45 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/09 03:35:45 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/09 03:35:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:35:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:35:50 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 03:35:55 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 03:36:00 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 03:36:05 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 03:36:05 App: TestDataTypes got un-deployed
2021/12/09 03:36:05 No eventing-con process running
2021/12/09 03:36:05 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/09 03:36:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:36:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 03:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:36:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:36:29 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/09 03:36:34 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/09 03:36:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.05s)
=== RUN   TestEnoentGet
2021/12/09 03:36:34 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:36:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/09 03:36:40 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:36:45 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:36:50 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:36:55 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:36:55 App: TestEnoentGet got deployed
2021/12/09 03:36:55 Starting bucket ops to source bucket
2021/12/09 03:37:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:37:00 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:37:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:37:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:37:06 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:37:11 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:37:16 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:37:21 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:37:21 App: TestEnoentGet got un-deployed
2021/12/09 03:37:21 No eventing-con process running
2021/12/09 03:37:21 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/09 03:37:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:37:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:37:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:37:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:37:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:37:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:37:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:37:54 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:37:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/09 03:38:00 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:38:05 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:38:10 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:38:15 Waiting for app: TestEnoentGet to get deployed
2021/12/09 03:38:15 App: TestEnoentGet got deployed
2021/12/09 03:38:15 Starting bucket ops to source bucket
2021/12/09 03:38:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:38:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:38:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:38:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:38:26 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:38:31 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:38:36 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:38:41 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 03:38:41 App: TestEnoentGet got un-deployed
2021/12/09 03:38:41 No eventing-con process running
2021/12/09 03:38:41 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/09 03:38:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:38:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:38:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:38:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:39:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:39:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:39:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.42s)
=== RUN   TestEnoentDelete
2021/12/09 03:39:14 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:39:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 03:39:20 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:39:26 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:39:31 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:39:36 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:39:36 App: TestEnoentDelete got deployed
2021/12/09 03:39:36 Starting bucket ops to source bucket
2021/12/09 03:39:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:39:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:39:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:39:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:39:46 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:39:51 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:39:56 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:40:01 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:40:01 App: TestEnoentDelete got un-deployed
2021/12/09 03:40:01 No eventing-con process running
2021/12/09 03:40:01 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/09 03:40:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:40:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:40:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:40:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:40:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:40:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:40:34 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:40:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 03:40:41 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:40:46 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:40:51 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:40:56 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 03:40:56 App: TestEnoentDelete got deployed
2021/12/09 03:40:56 Starting bucket ops to source bucket
2021/12/09 03:41:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:41:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:41:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:41:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:41:06 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:41:11 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:41:16 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:41:21 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 03:41:21 App: TestEnoentDelete got un-deployed
2021/12/09 03:41:21 No eventing-con process running
2021/12/09 03:41:21 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/09 03:41:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:41:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:41:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:41:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:41:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:41:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:41:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 03:41:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.18s)
=== RUN   TestError
2021/12/09 03:42:05 Waiting for app: TestError to get un-deployed
2021/12/09 03:42:05 App: TestError got un-deployed
2021/12/09 03:42:05 No eventing-con process running
2021/12/09 03:42:05 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/09 03:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:42:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:42:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:42:18 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:42:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/09 03:42:24 Waiting for app: TestError to get deployed
2021/12/09 03:42:29 Waiting for app: TestError to get deployed
2021/12/09 03:42:34 Waiting for app: TestError to get deployed
2021/12/09 03:42:39 Waiting for app: TestError to get deployed
2021/12/09 03:42:39 App: TestError got deployed
2021/12/09 03:42:39 Starting bucket ops to source bucket
2021/12/09 03:42:44 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/09 03:42:49 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/09 03:42:54 Waiting for dst bucket item count to get to: 300 curr count: 156
2021/12/09 03:42:54 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/09 03:42:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:42:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:43:00 Waiting for app: TestError to get un-deployed
2021/12/09 03:43:05 Waiting for app: TestError to get un-deployed
2021/12/09 03:43:10 Waiting for app: TestError to get un-deployed
2021/12/09 03:43:15 Waiting for app: TestError to get un-deployed
2021/12/09 03:43:15 App: TestError got un-deployed
2021/12/09 03:43:15 No eventing-con process running
2021/12/09 03:43:15 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/09 03:43:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:43:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:43:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:43:38 Waiting for dst bucket item count to get to: 0 curr count: 237
2021/12/09 03:43:43 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/09 03:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.76s)
=== RUN   TestCRLF
2021/12/09 03:43:53 Waiting for app: TestCRLF to get un-deployed
2021/12/09 03:43:53 App: TestCRLF got un-deployed
2021/12/09 03:43:53 No eventing-con process running
2021/12/09 03:43:53 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/09 03:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:44:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:44:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:44:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:44:07 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:44:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/09 03:44:13 Waiting for app: TestCRLF to get deployed
2021/12/09 03:44:18 Waiting for app: TestCRLF to get deployed
2021/12/09 03:44:23 Waiting for app: TestCRLF to get deployed
2021/12/09 03:44:28 Waiting for app: TestCRLF to get deployed
2021/12/09 03:44:28 App: TestCRLF got deployed
2021/12/09 03:44:28 Starting bucket ops to source bucket
2021/12/09 03:44:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 03:44:38 Waiting for dst bucket item count to get to: 100 curr count: 54
2021/12/09 03:44:43 Waiting for dst bucket item count to get to: 100 curr count: 54
2021/12/09 03:44:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 03:44:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:44:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:44:49 Waiting for app: TestCRLF to get un-deployed
2021/12/09 03:44:54 Waiting for app: TestCRLF to get un-deployed
2021/12/09 03:44:59 Waiting for app: TestCRLF to get un-deployed
2021/12/09 03:45:04 Waiting for app: TestCRLF to get un-deployed
2021/12/09 03:45:04 App: TestCRLF got un-deployed
2021/12/09 03:45:04 No eventing-con process running
2021/12/09 03:45:04 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/09 03:45:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:45:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:45:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:45:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:45:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 03:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (103.86s)
=== RUN   TestImportExport
2021/12/09 03:45:37 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 03:45:37 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 03:45:37 No eventing-con process running
2021/12/09 03:45:37 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/09 03:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:45:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:45:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:45:56 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 03:45:56 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 03:45:56 No eventing-con process running
2021/12/09 03:45:56 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/09 03:45:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:46:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:46:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:46:09 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:46:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 03:46:11 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:46:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 03:46:17 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 03:46:22 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 03:46:27 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 03:46:32 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 03:46:32 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/09 03:46:37 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 03:46:42 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 03:46:47 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 03:46:47 App: TestImportExport_bucket_op_on_update got deployed
2021/12/09 03:46:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:46:52 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 03:46:57 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 03:47:02 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 03:47:07 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 03:47:07 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 03:47:07 No eventing-con process running
2021/12/09 03:47:07 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/09 03:47:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:47:12 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 03:47:17 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 03:47:22 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 03:47:27 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 03:47:27 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 03:47:27 No eventing-con process running
2021/12/09 03:47:27 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/09 03:47:45 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 03:47:45 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 03:47:45 No eventing-con process running
2021/12/09 03:47:45 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/09 03:47:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:47:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:47:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:47:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:48:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 03:48:03 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 03:48:03 No eventing-con process running
2021/12/09 03:48:03 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/09 03:48:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:48:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:48:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:48:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.59s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/09 03:48:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:48:27 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 03:48:27 No eventing-con process running
2021/12/09 03:48:27 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/09 03:48:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:48:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:48:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:48:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:48:40 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:48:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 03:48:41 Starting bucket ops to source bucket
2021/12/09 03:48:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:48:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:48:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:49:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:49:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:49:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 03:49:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 03:49:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:49:09 Undeploying app: bucket_op_on_update
2021/12/09 03:49:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:49:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:49:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:49:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:49:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:49:30 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 03:49:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:49:30 No eventing-con process running
2021/12/09 03:49:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:49:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:50:14 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:50:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 03:50:15 Starting bucket ops to source bucket
2021/12/09 03:50:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:50:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:50:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:50:38 Waiting for dst bucket item count to get to: 5000 curr count: 3613
2021/12/09 03:50:43 Waiting for dst bucket item count to get to: 5000 curr count: 3613
2021/12/09 03:50:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:50:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 03:50:48 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 03:50:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:50:48 Undeploying app: bucket_op_on_update
2021/12/09 03:50:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:50:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:50:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:51:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:51:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:51:08 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 03:51:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:51:08 No eventing-con process running
2021/12/09 03:51:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:51:52 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:51:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 03:51:53 Starting bucket ops to source bucket
2021/12/09 03:52:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:52:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:52:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:52:16 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 03:52:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:52:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 03:52:21 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 03:52:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:52:21 Undeploying app: bucket_op_on_update
2021/12/09 03:52:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:52:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:52:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:52:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:52:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:52:41 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 03:52:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:52:41 No eventing-con process running
2021/12/09 03:52:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:52:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:53:25 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:53:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 03:53:26 Starting bucket ops to source bucket
2021/12/09 03:53:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:53:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:53:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:53:49 Waiting for dst bucket item count to get to: 5000 curr count: 1595
2021/12/09 03:53:54 Waiting for dst bucket item count to get to: 5000 curr count: 1595
2021/12/09 03:53:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:53:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 03:53:59 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 03:53:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:53:59 Undeploying app: bucket_op_on_update
2021/12/09 03:53:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:54:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:54:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:54:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:54:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:54:20 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 03:54:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:54:20 No eventing-con process running
2021/12/09 03:54:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:54:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:55:03 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:55:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 03:55:05 Starting bucket ops to source bucket
2021/12/09 03:55:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:55:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:55:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:55:27 Waiting for dst bucket item count to get to: 5000 curr count: 498
2021/12/09 03:55:32 Waiting for dst bucket item count to get to: 5000 curr count: 2998
2021/12/09 03:55:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:55:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 03:55:37 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 03:55:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:55:37 Undeploying app: bucket_op_on_update
2021/12/09 03:55:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:55:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:55:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:55:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:55:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 03:55:58 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 03:55:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:55:58 No eventing-con process running
2021/12/09 03:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:56:42 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (504.96s)
=== RUN   TestOnUpdateN1QLOp
2021/12/09 03:56:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 03:56:52 App: TestOnUpdateN1QLOp got un-deployed
2021/12/09 03:56:52 No eventing-con process running
2021/12/09 03:56:52 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/09 03:56:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:57:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:57:05 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:57:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/09 03:57:06 Starting bucket ops to source bucket
2021/12/09 03:57:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:57:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:57:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:57:29 Waiting for dst bucket item count to get to: 5000 curr count: 411
2021/12/09 03:57:34 Waiting for dst bucket item count to get to: 5000 curr count: 411
2021/12/09 03:57:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:57:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:57:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:57:40 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 03:57:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 03:57:50 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 03:57:55 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 03:57:55 App: TestOnUpdateN1QLOp got un-deployed
2021/12/09 03:57:55 No eventing-con process running
2021/12/09 03:57:55 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/09 03:57:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:58:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 03:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:58:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:58:18 Waiting for dst bucket item count to get to: 0 curr count: 3937
2021/12/09 03:58:23 Waiting for dst bucket item count to get to: 0 curr count: 1422
2021/12/09 03:58:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.51s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/09 03:58:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 03:58:33 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/09 03:58:33 No eventing-con process running
2021/12/09 03:58:33 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/09 03:58:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:58:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:58:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:58:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:58:47 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 03:58:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 03:58:48 Starting bucket ops to source bucket
2021/12/09 03:58:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:59:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:59:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 03:59:11 Waiting for dst bucket item count to get to: 5000 curr count: 538
2021/12/09 03:59:16 Waiting for dst bucket item count to get to: 5000 curr count: 538
2021/12/09 03:59:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 03:59:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 03:59:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 03:59:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 03:59:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 03:59:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 03:59:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 03:59:36 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/09 03:59:36 No eventing-con process running
2021/12/09 03:59:36 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/09 03:59:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 03:59:48 Waiting for dst bucket item count to get to: 0 curr count: 2958
2021/12/09 03:59:53 Waiting for dst bucket item count to get to: 0 curr count: 458
2021/12/09 03:59:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 03:59:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:00:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:00:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.45s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/09 04:00:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 04:00:15 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 04:00:15 No eventing-con process running
2021/12/09 04:00:15 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/09 04:00:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:00:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:00:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:00:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:00:28 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:00:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 04:00:29 Starting bucket ops to source bucket
2021/12/09 04:00:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:00:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:00:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:00:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:00:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 04:00:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:00:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:00:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:01:02 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 04:01:07 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 04:01:12 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 04:01:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 04:01:17 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 04:01:18 No eventing-con process running
2021/12/09 04:01:18 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/09 04:01:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:01:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:01:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 04:01:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:01:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:01:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:01:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.37s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/09 04:01:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 04:01:56 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/09 04:01:56 No eventing-con process running
2021/12/09 04:01:56 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/09 04:01:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:02:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:02:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:02:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:02:10 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:02:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/09 04:02:11 Starting bucket ops to source bucket
2021/12/09 04:02:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:02:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:02:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:02:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:02:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:02:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:02:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:02:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 04:02:44 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 04:02:49 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 04:02:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 04:02:54 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/09 04:02:54 No eventing-con process running
2021/12/09 04:02:54 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/09 04:02:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:03:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:03:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:03:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:03:18 Waiting for dst bucket item count to get to: 0 curr count: 2764
2021/12/09 04:03:23 Waiting for dst bucket item count to get to: 0 curr count: 264
2021/12/09 04:03:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.67s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/09 04:03:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 04:03:33 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/09 04:03:33 No eventing-con process running
2021/12/09 04:03:33 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/09 04:03:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:03:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:03:46 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:03:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/09 04:03:48 Starting bucket ops to source bucket
2021/12/09 04:03:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:04:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:04:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:04:11 Waiting for dst bucket item count to get to: 5000 curr count: 605
2021/12/09 04:04:16 Waiting for dst bucket item count to get to: 5000 curr count: 605
2021/12/09 04:04:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:04:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:04:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:04:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 04:04:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 04:04:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 04:04:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 04:04:36 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/09 04:04:36 No eventing-con process running
2021/12/09 04:04:36 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/09 04:04:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:04:48 Waiting for dst bucket item count to get to: 0 curr count: 2977
2021/12/09 04:04:53 Waiting for dst bucket item count to get to: 0 curr count: 477
2021/12/09 04:04:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:04:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:05:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:05:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (101.96s)
=== RUN   TestOnDeleteBucketOp
2021/12/09 04:05:15 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 04:05:15 App: TestOnDeleteBucketOp got un-deployed
2021/12/09 04:05:15 No eventing-con process running
2021/12/09 04:05:15 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/09 04:05:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:05:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:05:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:05:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:05:28 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:05:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 04:05:29 Starting bucket ops to source bucket
2021/12/09 04:05:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:05:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:05:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:05:55 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/09 04:05:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:05:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:05:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:06:00 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 04:06:05 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 04:06:10 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 04:06:15 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 04:06:15 App: TestOnDeleteBucketOp got un-deployed
2021/12/09 04:06:15 No eventing-con process running
2021/12/09 04:06:15 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/09 04:06:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:06:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:06:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:06:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:06:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.87s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/09 04:06:45 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 04:06:45 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/09 04:06:45 No eventing-con process running
2021/12/09 04:06:45 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/09 04:06:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:06:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:06:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:06:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:06:58 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:06:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/09 04:06:59 Starting bucket ops to source bucket
2021/12/09 04:07:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:07:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:07:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:07:25 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/09 04:07:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:07:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:07:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:07:30 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 04:07:35 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 04:07:40 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 04:07:45 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 04:07:45 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/09 04:07:45 No eventing-con process running
2021/12/09 04:07:45 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/09 04:07:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:07:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:07:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:08:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:08:09 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/09 04:08:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (94.62s)
=== RUN   TestTimerBucketOp
2021/12/09 04:08:19 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 04:08:19 App: TestTimerBucketOp got un-deployed
2021/12/09 04:08:19 No eventing-con process running
2021/12/09 04:08:19 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/09 04:08:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:08:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:08:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:08:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:08:33 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:08:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 04:08:34 Starting bucket ops to source bucket
2021/12/09 04:08:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:08:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:08:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:08:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:09:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:09:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:09:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:09:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 04:09:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:09:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:09:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:09:22 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 04:09:27 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 04:09:32 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 04:09:37 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 04:09:37 App: TestTimerBucketOp got un-deployed
2021/12/09 04:09:37 No eventing-con process running
2021/12/09 04:09:37 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/09 04:09:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:09:49 Waiting for dst bucket item count to get to: 0 curr count: 4833
2021/12/09 04:09:54 Waiting for dst bucket item count to get to: 0 curr count: 2333
2021/12/09 04:09:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:09:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:10:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:10:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.34s)
=== RUN   TestTimerInPastBucketOp
2021/12/09 04:10:16 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 04:10:16 App: TestTimerInPastBucketOp got un-deployed
2021/12/09 04:10:16 No eventing-con process running
2021/12/09 04:10:16 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/09 04:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:10:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:10:29 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:10:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 04:10:30 Starting bucket ops to source bucket
2021/12/09 04:10:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:10:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:10:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:10:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:10:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:11:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:11:08 Waiting for dst bucket item count to get to: 5000 curr count: 3179
2021/12/09 04:11:13 Waiting for dst bucket item count to get to: 5000 curr count: 3179
2021/12/09 04:11:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:11:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:11:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:11:18 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 04:11:23 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 04:11:28 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 04:11:33 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 04:11:33 App: TestTimerInPastBucketOp got un-deployed
2021/12/09 04:11:33 No eventing-con process running
2021/12/09 04:11:33 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/09 04:11:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:11:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:11:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:11:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:11:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.39s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/09 04:12:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:12:07 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 04:12:07 No eventing-con process running
2021/12/09 04:12:07 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/09 04:12:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:12:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:12:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:12:20 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:12:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 04:12:22 Starting bucket ops to source bucket
2021/12/09 04:12:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:12:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:12:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:12:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:12:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:12:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:12:59 Waiting for dst bucket item count to get to: 5000 curr count: 1464
2021/12/09 04:13:04 Waiting for dst bucket item count to get to: 5000 curr count: 1464
2021/12/09 04:13:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:13:09 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 04:13:09 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 04:13:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:13:09 Undeploying app: bucket_op_with_timer
2021/12/09 04:13:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:13:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:13:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:13:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:13:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:13:30 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 04:13:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:13:30 No eventing-con process running
2021/12/09 04:13:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:13:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:14:14 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:14:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 04:14:15 Starting bucket ops to source bucket
2021/12/09 04:14:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:14:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:14:58 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 04:14:58 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 04:14:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:14:58 Undeploying app: bucket_op_with_timer
2021/12/09 04:14:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:15:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:15:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:15:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:15:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:15:18 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 04:15:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:15:18 No eventing-con process running
2021/12/09 04:15:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:15:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:16:02 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:16:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 04:16:03 Starting bucket ops to source bucket
2021/12/09 04:16:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:16:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:16:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:16:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:16:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:16:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:16:41 Waiting for dst bucket item count to get to: 5000 curr count: 75
2021/12/09 04:16:46 Waiting for dst bucket item count to get to: 5000 curr count: 2575
2021/12/09 04:16:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:16:51 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 04:16:51 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 04:16:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:16:51 Undeploying app: bucket_op_with_timer
2021/12/09 04:16:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:16:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:17:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:17:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:17:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:17:11 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 04:17:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:17:11 No eventing-con process running
2021/12/09 04:17:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:17:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:17:55 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:17:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 04:17:56 Starting bucket ops to source bucket
2021/12/09 04:18:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:18:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:18:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:18:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:18:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:18:29 Waiting for dst bucket item count to get to: 5000 curr count: 786
2021/12/09 04:18:34 Waiting for dst bucket item count to get to: 5000 curr count: 786
2021/12/09 04:18:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:18:39 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 04:18:39 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 04:18:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:18:39 Undeploying app: bucket_op_with_timer
2021/12/09 04:18:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:18:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:18:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:18:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:18:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:18:59 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 04:18:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:18:59 No eventing-con process running
2021/12/09 04:19:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:19:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:19:43 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:19:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 04:19:45 Starting bucket ops to source bucket
2021/12/09 04:19:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:19:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:20:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:20:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:20:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:20:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:20:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:20:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:20:27 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 04:20:27 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 04:20:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:20:27 Undeploying app: bucket_op_with_timer
2021/12/09 04:20:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:20:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:20:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:20:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:20:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 04:20:48 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 04:20:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:20:48 No eventing-con process running
2021/12/09 04:20:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:20:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:21:32 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (574.86s)
=== RUN   TestMultipleHandlers
2021/12/09 04:21:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 04:21:42 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/09 04:21:42 No eventing-con process running
2021/12/09 04:21:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/09 04:21:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:21:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:21:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:22:00 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 04:22:00 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/09 04:22:00 No eventing-con process running
2021/12/09 04:22:00 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/09 04:22:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:22:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:22:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:22:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:22:14 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:22:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 04:22:15 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:22:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/09 04:22:21 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 04:22:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 04:22:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 04:22:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 04:22:36 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/09 04:22:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 04:22:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 04:22:51 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 04:22:51 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/09 04:22:51 Starting bucket ops to source bucket
2021/12/09 04:23:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:23:01 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:23:01 Starting bucket ops to source bucket
2021/12/09 04:23:16 Waiting for dst bucket item count to get to: 10000 curr count: 7059
2021/12/09 04:23:21 Waiting for dst bucket item count to get to: 10000 curr count: 7059
2021/12/09 04:23:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 04:23:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:23:21 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:23:21 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:23:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:23:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 04:23:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 04:23:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 04:23:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 04:23:41 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/09 04:23:42 No eventing-con process running
2021/12/09 04:23:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/09 04:23:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:23:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:24:01 Waiting for dst bucket item count to get to: 0 curr count: 7582
2021/12/09 04:24:06 Waiting for dst bucket item count to get to: 0 curr count: 7582
2021/12/09 04:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:24:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:24:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 04:24:16 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 04:24:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 04:24:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 04:24:26 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/09 04:24:26 No eventing-con process running
2021/12/09 04:24:26 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/09 04:24:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:24:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:24:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:24:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.61s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/09 04:24:49 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 04:24:49 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/09 04:24:49 No eventing-con process running
2021/12/09 04:24:49 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/09 04:24:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:24:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:24:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:25:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:25:03 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:25:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 04:25:04 Starting bucket ops to source bucket
2021/12/09 04:25:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:25:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:25:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:25:27 Waiting for dst bucket item count to get to: 5000 curr count: 2153
2021/12/09 04:25:32 Waiting for dst bucket item count to get to: 5000 curr count: 4653
2021/12/09 04:25:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:25:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:25:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 04:25:37 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 04:25:37 Pausing app: bucket_op_on_update
2021/12/09 04:25:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:25:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 04:25:42 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 04:25:42 Resuming app: bucket_op_on_update
2021/12/09 04:25:43 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:25:43 Starting bucket ops to source bucket
2021/12/09 04:25:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:25:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:26:01 Waiting for dst bucket item count to get to: 10000 curr count: 5007
2021/12/09 04:26:06 Waiting for dst bucket item count to get to: 10000 curr count: 7507
2021/12/09 04:26:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 04:26:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:26:11 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 04:26:11 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 04:26:11 Pausing app: bucket_op_on_update
2021/12/09 04:26:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:26:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 04:26:16 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 04:26:16 Resuming app: bucket_op_on_update
2021/12/09 04:26:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:26:17 Starting bucket ops to source bucket
2021/12/09 04:26:25 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 04:26:30 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 04:26:35 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/09 04:26:35 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/09 04:26:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:26:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 04:26:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 04:26:40 Pausing app: bucket_op_on_update
2021/12/09 04:26:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:26:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 04:26:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 04:26:45 Resuming app: bucket_op_on_update
2021/12/09 04:26:47 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:26:47 Starting bucket ops to source bucket
2021/12/09 04:26:55 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:27:00 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:27:05 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:27:05 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 04:27:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:27:10 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 04:27:10 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 04:27:10 Pausing app: bucket_op_on_update
2021/12/09 04:27:10 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:27:15 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 04:27:15 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 04:27:15 Resuming app: bucket_op_on_update
2021/12/09 04:27:16 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:27:16 Starting bucket ops to source bucket
2021/12/09 04:27:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:27:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:27:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:27:34 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 04:27:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:27:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 04:27:39 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 04:27:39 Pausing app: bucket_op_on_update
2021/12/09 04:27:39 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:27:44 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 04:27:44 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 04:27:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:27:49 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 04:27:54 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 04:27:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 04:28:04 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 04:28:04 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/09 04:28:04 No eventing-con process running
2021/12/09 04:28:04 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/09 04:28:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:28:16 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 04:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:28:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:28:28 Waiting for dst bucket item count to get to: 0 curr count: 13022
2021/12/09 04:28:33 Waiting for dst bucket item count to get to: 0 curr count: 522
2021/12/09 04:28:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.33s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/09 04:28:43 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 04:28:43 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/09 04:28:43 No eventing-con process running
2021/12/09 04:28:43 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/09 04:28:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:28:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:28:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:28:56 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:28:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 04:28:57 Starting bucket ops to source bucket
2021/12/09 04:29:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:29:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:29:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:29:40 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 04:29:40 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 04:29:40 Pausing app: bucket_op_with_timer
2021/12/09 04:29:40 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:29:45 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 04:29:45 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 04:29:45 Resuming app: bucket_op_with_timer
2021/12/09 04:29:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:29:47 Starting bucket ops to source bucket
2021/12/09 04:29:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:29:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:30:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:30:09 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:30:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:30:20 Waiting for dst bucket item count to get to: 10000 curr count: 7196
2021/12/09 04:30:25 Waiting for dst bucket item count to get to: 10000 curr count: 9696
2021/12/09 04:30:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 04:30:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:30:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 04:30:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 04:30:30 Pausing app: bucket_op_with_timer
2021/12/09 04:30:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:30:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 04:30:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 04:30:35 Resuming app: bucket_op_with_timer
2021/12/09 04:30:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:30:36 Starting bucket ops to source bucket
2021/12/09 04:30:44 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 04:30:49 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 04:30:54 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 04:30:59 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 04:31:04 Waiting for dst bucket item count to get to: 15000 curr count: 10330
2021/12/09 04:31:09 Waiting for dst bucket item count to get to: 15000 curr count: 12167
2021/12/09 04:31:14 Waiting for dst bucket item count to get to: 15000 curr count: 14337
2021/12/09 04:31:14 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/09 04:31:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:31:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 04:31:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 04:31:19 Pausing app: bucket_op_with_timer
2021/12/09 04:31:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:31:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 04:31:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 04:31:24 Resuming app: bucket_op_with_timer
2021/12/09 04:31:25 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:31:25 Starting bucket ops to source bucket
2021/12/09 04:31:33 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:31:38 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:31:43 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:31:48 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:31:53 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 04:31:58 Waiting for dst bucket item count to get to: 20000 curr count: 16471
2021/12/09 04:32:03 Waiting for dst bucket item count to get to: 20000 curr count: 17927
2021/12/09 04:32:08 Waiting for dst bucket item count to get to: 20000 curr count: 18956
2021/12/09 04:32:08 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 04:32:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:32:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 04:32:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 04:32:13 Pausing app: bucket_op_with_timer
2021/12/09 04:32:13 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:32:18 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 04:32:18 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 04:32:18 Resuming app: bucket_op_with_timer
2021/12/09 04:32:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:32:19 Starting bucket ops to source bucket
2021/12/09 04:32:27 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:32:32 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:32:37 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:32:42 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:32:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:32:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 04:32:52 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 04:32:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:32:57 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 04:32:57 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 04:32:57 Pausing app: bucket_op_with_timer
2021/12/09 04:32:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:33:03 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 04:33:03 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 04:33:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:33:08 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 04:33:13 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 04:33:18 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 04:33:23 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 04:33:23 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/09 04:33:23 No eventing-con process running
2021/12/09 04:33:23 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/09 04:33:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:33:34 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 04:33:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:33:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:33:46 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 04:33:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (313.58s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/09 04:33:56 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 04:33:56 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/09 04:33:56 No eventing-con process running
2021/12/09 04:33:56 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/09 04:33:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:34:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:34:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:34:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:34:10 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:34:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/09 04:34:16 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 04:34:21 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 04:34:26 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 04:34:31 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 04:34:31 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/09 04:34:31 Starting bucket ops to source bucket
2021/12/09 04:34:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:34:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:34:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:34:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:34:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:34:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:34:55 Pausing app: bucket_op_with_timer
2021/12/09 04:34:55 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:35:00 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/09 04:35:00 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/09 04:35:00 Resuming app: bucket_op_with_timer
2021/12/09 04:35:02 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:35:02 Starting bucket ops to source bucket
2021/12/09 04:35:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:35:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:35:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:35:28 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:35:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:35:38 Waiting for dst bucket item count to get to: 10000 curr count: 6669
2021/12/09 04:35:43 Waiting for dst bucket item count to get to: 10000 curr count: 6669
2021/12/09 04:35:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 04:35:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:35:48 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 04:35:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 04:35:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 04:36:03 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 04:36:03 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/09 04:36:03 No eventing-con process running
2021/12/09 04:36:03 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/09 04:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:36:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 04:36:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:36:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 04:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (160.58s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/09 04:36:37 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 04:36:37 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/09 04:36:37 No eventing-con process running
2021/12/09 04:36:37 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/09 04:36:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:36:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:36:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:36:50 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:36:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 04:36:57 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 04:37:02 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 04:37:07 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 04:37:12 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 04:37:12 App: TestPauseResumeWithEventingReb got deployed
2021/12/09 04:37:12 Starting bucket ops to source bucket
2021/12/09 04:37:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:37:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:37:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:37:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:37:41 Waiting for dst bucket item count to get to: 5000 curr count: 4878
2021/12/09 04:37:46 Waiting for dst bucket item count to get to: 5000 curr count: 4878
2021/12/09 04:37:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:37:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:37:46 Pausing app: bucket_op_with_timer
2021/12/09 04:37:46 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:37:51 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/09 04:37:51 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/09 04:37:51 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 04:37:51 Removing node(s): [] from the cluster
2021/12/09 04:37:57 Rebalance progress: 29.16666666666666
2021/12/09 04:38:02 Rebalance progress: 29.16666666666666
2021/12/09 04:38:07 Rebalance progress: 100
2021/12/09 04:38:07 Resuming app: bucket_op_with_timer
2021/12/09 04:38:08 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:38:08 Starting bucket ops to source bucket
2021/12/09 04:38:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:38:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:38:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:38:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 04:38:39 Waiting for dst bucket item count to get to: 10000 curr count: 5230
2021/12/09 04:38:44 Waiting for dst bucket item count to get to: 10000 curr count: 5230
2021/12/09 04:38:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 04:38:44 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 04:38:49 Rebalance progress: 29.16666666666666
2021/12/09 04:38:54 Rebalance progress: 29.16666666666666
2021/12/09 04:38:59 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/09 04:39:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:39:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:39:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:39:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:40:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:40:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:40:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:40:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:41:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:41:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:41:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 04:41:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:76} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 5f4908465ee0d5807728bfbde7abcbfb
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: fadc418947a40bccbf1c26f2e6b60e8d
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
Producer node: 127.0.0.1:9303	Node UUID: 3f5f36afcbd7e116a4f34075d21d037d
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]


2021/12/09 04:42:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:42:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:42:10 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 04:42:15 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 04:42:20 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 04:42:25 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 04:42:25 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/09 04:42:25 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:42:25 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/09 04:42:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:42:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 04:42:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:42:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:42:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8845
2021/12/09 04:42:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3845
2021/12/09 04:42:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (386.74s)
=== RUN   TestBucketDeleteAfterPause
2021/12/09 04:43:04 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/09 04:43:04 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/09 04:43:04 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:43:04 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/09 04:43:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:43:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:43:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:43:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:43:17 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:43:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/09 04:43:23 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 04:43:28 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 04:43:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 04:43:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 04:43:38 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/09 04:43:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:43:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 456
2021/12/09 04:43:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2956
2021/12/09 04:43:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2956
2021/12/09 04:43:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:43:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:43:58 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/09 04:43:58 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:44:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/09 04:44:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/09 04:44:03 handler_test.go:716: Deleting source bucket: default
2021/12/09 04:44:09 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/09 04:44:09 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/09 04:44:15 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/09 04:44:15 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/09 04:44:15 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:44:15 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/09 04:44:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:44:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:44:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:44:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:44:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2671
2021/12/09 04:44:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 171
2021/12/09 04:44:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (109.11s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/09 04:44:53 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 04:44:53 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/09 04:44:53 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:44:53 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/09 04:44:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:44:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:45:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:45:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:45:06 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:45:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/09 04:45:12 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 04:45:17 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 04:45:22 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 04:45:27 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 04:45:27 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/09 04:45:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:46:02 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/09 04:46:02 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:46:07 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/09 04:46:07 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/09 04:49:07 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/09 04:49:07 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/09 04:49:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/09 04:49:13 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 04:49:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:49:13 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/09 04:49:18 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 04:49:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:49:18 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2970 expected 5000
2021/12/09 04:49:23 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 04:49:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:49:23 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/09 04:49:28 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 04:49:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:49:28 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/09 04:49:28 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/09 04:49:28 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:49:38 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 04:49:43 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 04:49:48 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 04:49:48 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/09 04:49:48 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:49:48 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/09 04:49:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:50:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 04:50:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 04:50:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:50:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:50:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.78s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/09 04:50:21 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 04:50:21 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/09 04:50:22 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:50:22 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/09 04:50:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:50:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:50:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:50:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:50:35 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:50:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 04:50:41 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:50:46 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:50:51 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:50:56 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:50:56 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/09 04:50:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:51:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:51:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3089
2021/12/09 04:51:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3089
2021/12/09 04:51:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:51:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:51:15 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/09 04:51:15 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:51:45 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/09 04:51:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 04:51:51 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:51:56 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:52:01 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:52:06 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 04:52:06 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/09 04:52:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:52:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:52:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:52:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:52:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:52:24 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/09 04:52:24 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:52:34 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 04:52:39 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 04:52:44 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 04:52:44 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/09 04:52:44 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:52:44 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/09 04:52:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:52:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:52:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:52:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.64s)
=== RUN   TestCPPWorkerCleanup
2021/12/09 04:53:07 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 04:53:07 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/09 04:53:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:53:07 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/09 04:53:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:53:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:53:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:53:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:53:20 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:53:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/09 04:53:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:53:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:53:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:53:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:53:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:53:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3775
2021/12/09 04:53:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3775
2021/12/09 04:53:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:53:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:53:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:53:59 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 04:54:04 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 04:54:09 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 04:54:14 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 04:54:14 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/09 04:54:14 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:54:14 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/09 04:54:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:54:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:54:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:54:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:54:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:54:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.62s)
=== RUN   TestWithUserXattrs
2021/12/09 04:55:17 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 04:55:17 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/09 04:55:17 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:55:17 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/09 04:55:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:55:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:55:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:55:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:55:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:55:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:55:34 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:55:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/09 04:55:40 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 04:55:45 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 04:55:50 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 04:55:55 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 04:55:55 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/09 04:55:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:56:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:56:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3855
2021/12/09 04:56:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3855
2021/12/09 04:56:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:56:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:56:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 04:56:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 04:56:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 04:56:29 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 04:56:34 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 04:56:34 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/09 04:56:34 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:56:34 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/09 04:56:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:56:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:56:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:56:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:56:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 04:56:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.63s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/09 04:57:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 04:57:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 04:57:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:57:07 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/09 04:57:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:57:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:57:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:57:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:57:20 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/09 04:57:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/09 04:57:27 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 04:57:32 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 04:57:37 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 04:57:42 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 04:57:42 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/09 04:57:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:57:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:57:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 04:57:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 04:57:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:57:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:58:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1075
2021/12/09 04:58:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1075
2021/12/09 04:58:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:58:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:58:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 04:58:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:58:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 04:58:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14414
2021/12/09 04:58:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14414
2021/12/09 04:58:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 04:58:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 04:58:57 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 04:59:02 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 04:59:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 04:59:12 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 04:59:17 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 04:59:17 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 04:59:17 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:59:22 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 04:59:22 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 04:59:22 eventingSetup.go:46: No eventing-con process running
2021/12/09 04:59:22 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/09 04:59:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:59:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 04:59:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/09 04:59:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 04:59:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 04:59:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 16647
2021/12/09 04:59:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 16647
2021/12/09 04:59:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.71s)
=== RUN   TestMetaBucketDelete
2021/12/09 05:00:05 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 05:00:05 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 05:00:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:00:05 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/09 05:00:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:00:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:00:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:00:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:00:18 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:00:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 05:00:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 05:00:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 05:00:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 05:00:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 05:00:39 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/09 05:00:39 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/09 05:00:41 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/09 05:00:46 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 05:00:46 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 05:01:01 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 05:01:01 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 05:01:01 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:01:01 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/09 05:01:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:01:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:01:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:01:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.65s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/09 05:01:24 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:01:24 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:01:24 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:01:24 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 05:01:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:01:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:01:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:01:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:01:37 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:01:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 05:01:57 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/09 05:01:57 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/09 05:02:12 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:02:12 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:02:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:02:27 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:02:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:02:27 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 05:02:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:02:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:02:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:02:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.71s)
=== RUN   TestSourceBucketDelete
2021/12/09 05:02:50 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 05:02:50 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 05:02:50 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:02:50 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/09 05:02:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:02:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:02:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:03:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:03:03 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:03:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 05:03:10 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 05:03:15 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 05:03:20 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 05:03:25 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 05:03:25 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/09 05:03:25 handler_test.go:945: Deleting source bucket: default
2021/12/09 05:03:26 handler_test.go:947: Deleted source bucket: default
2021/12/09 05:03:31 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 05:03:31 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 05:03:46 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 05:03:46 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 05:03:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:03:46 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/09 05:03:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:03:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:03:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:03:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.06s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/09 05:04:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:04:09 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:04:10 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:04:10 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 05:04:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:04:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:04:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:04:23 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:04:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 05:04:43 handler_test.go:966: Deleting source bucket: default
2021/12/09 05:04:44 handler_test.go:968: Deleted source bucket: default
2021/12/09 05:04:59 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:04:59 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:05:14 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:05:14 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:05:14 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:05:14 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 05:05:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:05:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:05:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:05:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.66s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/09 05:05:37 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:05:37 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:05:37 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:05:37 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 05:05:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:05:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:05:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:05:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:05:50 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:05:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 05:06:10 handler_test.go:988: Deleting source bucket: default
2021/12/09 05:06:12 handler_test.go:990: Deleted source bucket: default
2021/12/09 05:06:12 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/09 05:06:12 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/09 05:06:27 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:06:27 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:06:42 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 05:06:42 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 05:06:42 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:06:42 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 05:06:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:06:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:06:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:06:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.15s)
=== RUN   TestUndeployDuringBootstrap
2021/12/09 05:07:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 05:07:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 05:07:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:07:05 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/09 05:07:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:07:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:07:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:07:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:07:18 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:07:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 05:07:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 05:07:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:07:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:07:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 05:07:31 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/09 05:07:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 05:07:36 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 05:07:41 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/09 05:07:46 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 05:07:52 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 05:07:57 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 05:07:57 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 05:07:57 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:08:22 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 05:08:22 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 05:08:22 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:08:22 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/09 05:08:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:08:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:08:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 05:08:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:08:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:08:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1340
2021/12/09 05:08:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1340
2021/12/09 05:08:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.27s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/09 05:09:00 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:09:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/09 05:09:06 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 05:09:11 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 05:09:16 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 05:09:21 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 05:09:21 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/09 05:09:21 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/09 05:09:21 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 05:09:21 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/09 05:09:26 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/09 05:09:26 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/09 05:09:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:09:26 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/09 05:09:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:09:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:09:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:09:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.58s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/09 05:09:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 05:09:49 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 05:09:49 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:09:49 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/09 05:09:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:09:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:10:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:10:02 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:10:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/09 05:10:09 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 05:10:14 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 05:10:19 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 05:10:24 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 05:10:24 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/09 05:10:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 05:10:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:10:59 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 05:11:04 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 05:11:09 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 05:11:14 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 05:11:14 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 05:11:14 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:12:59 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 05:12:59 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 05:12:59 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:12:59 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/09 05:13:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:13:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4561
2021/12/09 05:13:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4561
2021/12/09 05:13:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:13:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:13:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:13:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.00s)
=== RUN   TestUndeployWithKVFailover
2021/12/09 05:13:37 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 05:13:37 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/09 05:13:37 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:13:37 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/09 05:13:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:13:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:13:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:13:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:13:50 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:13:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/09 05:13:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 05:13:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:14:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:14:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:14:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:14:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:14:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:14:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1480
2021/12/09 05:14:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1480
2021/12/09 05:14:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 05:14:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 05:14:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 05:14:39 Rebalance progress: 29.16666666666666
2021/12/09 05:14:44 Rebalance progress: 29.16666666666666
2021/12/09 05:14:49 Rebalance progress: 100
2021/12/09 05:14:49 Failing over: [127.0.0.1:19003]
2021/12/09 05:14:59 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 05:15:59 Removing node(s): [] from the cluster
2021/12/09 05:16:04 Rebalance progress: 29.16666666666666
2021/12/09 05:16:09 Rebalance progress: 29.16666666666666
2021/12/09 05:16:14 Rebalance progress: 100
2021/12/09 05:16:19 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 05:16:19 App: TestUndeployWithKVFailover got un-deployed
2021/12/09 05:16:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:16:24 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 05:16:24 App: TestUndeployWithKVFailover got un-deployed
2021/12/09 05:16:24 No eventing-con process running
2021/12/09 05:16:24 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/09 05:16:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:16:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:16:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:16:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:16:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.88s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/09 05:16:57 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 05:16:57 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/09 05:16:57 No eventing-con process running
2021/12/09 05:16:57 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/09 05:16:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:17:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:17:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:17:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:17:10 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:17:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/09 05:17:16 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 05:17:21 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 05:17:26 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 05:17:31 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 05:17:31 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/09 05:17:31 Starting bucket ops to source bucket
2021/12/09 05:17:52 Waiting for dst bucket item count to get to: 20000 curr count: 8905
2021/12/09 05:17:52 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 05:17:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:17:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:18:04 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/09 05:18:09 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 05:18:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:18:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:18:21 Waiting for dst bucket item count to get to: 0 curr count: 18477
2021/12/09 05:18:26 Waiting for dst bucket item count to get to: 0 curr count: 18477
2021/12/09 05:18:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:18:26 Starting bucket ops to source bucket
2021/12/09 05:18:38 Waiting for dst bucket item count to get to: 10000 curr count: 3028
2021/12/09 05:18:43 Waiting for dst bucket item count to get to: 10000 curr count: 3028
2021/12/09 05:18:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 05:18:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:18:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:18:49 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 05:18:54 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 05:18:59 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 05:19:04 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 05:19:04 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/09 05:19:04 No eventing-con process running
2021/12/09 05:19:04 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/09 05:19:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:19:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 05:19:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:19:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:19:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 05:19:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (159.61s)
=== RUN   TestTimerOverwrite
2021/12/09 05:19:37 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 05:19:37 App: TestTimerOverwrite got un-deployed
2021/12/09 05:19:37 No eventing-con process running
2021/12/09 05:19:37 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/09 05:19:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:19:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:19:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:19:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:19:50 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:19:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/09 05:19:56 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 05:20:01 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 05:20:06 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 05:20:11 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 05:20:11 App: TestTimerOverwrite got deployed
2021/12/09 05:20:21 Starting bucket ops to source bucket
2021/12/09 05:20:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:20:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:20:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:20:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:20:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:20:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:21:30 Waiting for dst bucket item count to get to: 5000 curr count: 2836
2021/12/09 05:21:35 Waiting for dst bucket item count to get to: 5000 curr count: 2836
2021/12/09 05:21:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 05:21:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:21:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:21:50 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 05:21:55 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 05:22:00 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 05:22:05 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 05:22:05 App: TestTimerOverwrite got un-deployed
2021/12/09 05:22:05 No eventing-con process running
2021/12/09 05:22:05 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/09 05:22:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:22:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:22:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:22:28 Waiting for dst bucket item count to get to: 0 curr count: 3997
2021/12/09 05:22:33 Waiting for dst bucket item count to get to: 0 curr count: 1535
2021/12/09 05:22:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (186.66s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/09 05:22:38 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 05:22:39 Removing node(s): [] from the cluster
2021/12/09 05:22:44 Rebalance progress: 100
2021/12/09 05:22:44 Failing over: [127.0.0.1:19003]
2021/12/09 05:22:49 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 05:22:49 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 05:22:49 No eventing-con process running
2021/12/09 05:22:49 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/09 05:22:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:22:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:22:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:23:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:23:02 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:23:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 05:23:03 Starting bucket ops to source bucket
2021/12/09 05:23:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:23:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:23:13 Context: Post to set retry counter, response code: 200 dump: 
2021/12/09 05:23:13 set retry counter, response &{[] }
2021/12/09 05:24:18 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 05:24:18 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 05:24:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:24:18 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/09 05:24:23 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 05:24:23 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 05:24:23 No eventing-con process running
2021/12/09 05:24:23 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/09 05:24:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:24:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:24:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:24:46 Waiting for dst bucket item count to get to: 0 curr count: 2700
2021/12/09 05:24:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:24:46 Removing node(s): [] from the cluster
2021/12/09 05:24:51 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (137.95s)
=== RUN   TestOnUpdateSrcMutation
2021/12/09 05:25:01 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 05:25:01 App: TestOnUpdateSrcMutation got un-deployed
2021/12/09 05:25:01 No eventing-con process running
2021/12/09 05:25:01 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/09 05:25:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:25:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:25:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:25:14 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:25:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/09 05:25:16 Starting bucket ops to source bucket
2021/12/09 05:25:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 05:25:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:25:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:25:48 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 05:25:53 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 05:25:58 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 05:26:03 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 05:26:03 App: TestOnUpdateSrcMutation got un-deployed
2021/12/09 05:26:03 No eventing-con process running
2021/12/09 05:26:03 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/09 05:26:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:26:14 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 05:26:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:26:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:26:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:26:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.62s)
=== RUN   TestOnDeleteSrcMutation
2021/12/09 05:26:36 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 05:26:36 App: TestOnDeleteSrcMutation got un-deployed
2021/12/09 05:26:36 No eventing-con process running
2021/12/09 05:26:36 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/09 05:26:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:26:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:26:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:26:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:26:49 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:26:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/09 05:26:50 Starting bucket ops to source bucket
2021/12/09 05:27:19 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 05:27:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:27:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:27:24 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 05:27:29 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 05:27:34 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 05:27:39 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 05:27:39 App: TestOnDeleteSrcMutation got un-deployed
2021/12/09 05:27:39 No eventing-con process running
2021/12/09 05:27:39 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/09 05:27:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:27:50 Waiting for dst bucket item count to get to: 0 curr count: 3994
2021/12/09 05:27:55 Waiting for dst bucket item count to get to: 0 curr count: 3994
2021/12/09 05:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:27:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:28:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.10s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/09 05:28:12 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 05:28:12 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/09 05:28:12 No eventing-con process running
2021/12/09 05:28:12 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/09 05:28:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:28:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:28:25 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:28:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 05:28:26 Starting bucket ops to source bucket
2021/12/09 05:29:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 05:29:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:29:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:29:08 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 05:29:13 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 05:29:18 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 05:29:23 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 05:29:23 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/09 05:29:23 No eventing-con process running
2021/12/09 05:29:23 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/09 05:29:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:29:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 05:29:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:29:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:29:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.34s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/09 05:29:51 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 05:29:51 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/09 05:29:51 No eventing-con process running
2021/12/09 05:29:51 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/09 05:29:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:29:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:29:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:30:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:30:04 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:30:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 05:30:06 Starting bucket ops to source bucket
2021/12/09 05:30:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 05:30:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:30:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:30:49 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 05:30:54 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 05:30:59 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 05:31:04 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 05:31:04 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/09 05:31:04 No eventing-con process running
2021/12/09 05:31:05 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/09 05:31:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:31:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 05:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:31:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:31:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.28s)
=== RUN   TestInterHandlerRecursion
2021/12/09 05:31:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:31:33 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 05:31:33 No eventing-con process running
2021/12/09 05:31:33 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/09 05:31:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:31:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:31:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:31:51 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:31:51 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 05:31:51 No eventing-con process running
2021/12/09 05:31:51 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 05:31:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:31:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:32:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:32:04 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:32:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/09 05:32:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:32:10 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:32:15 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:32:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:32:25 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:32:25 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/09 05:32:25 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:32:25 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/09 05:32:25 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/09 05:32:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:32:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:32:35 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:32:40 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:32:45 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:32:45 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 05:32:45 No eventing-con process running
2021/12/09 05:32:45 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/09 05:32:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:32:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:32:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:32:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:33:03 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:33:03 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 05:33:03 No eventing-con process running
2021/12/09 05:33:03 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 05:33:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.51s)
=== RUN   TestInterBucketRecursion
2021/12/09 05:33:21 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:33:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 05:33:22 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:33:22 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:33:22 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/09 05:33:27 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 05:33:32 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 05:33:37 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 05:33:42 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 05:33:42 App: TestInterBucketRecursion_function1 got deployed
2021/12/09 05:33:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:33:47 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:33:52 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:33:57 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:34:02 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:34:02 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/09 05:34:02 No eventing-con process running
2021/12/09 05:34:03 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/09 05:34:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:34:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:34:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:34:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:34:20 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/09 05:34:20 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/09 05:34:21 No eventing-con process running
2021/12/09 05:34:21 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/09 05:34:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:34:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:34:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:34:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.48s)
=== RUN   TestLargeHandler
2021/12/09 05:34:34 Changed force_compress value to false
2021/12/09 05:34:34 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:34:34 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/09 05:34:34 Changed force_compress value to true
2021/12/09 05:34:34 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:34:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/09 05:34:41 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 05:34:46 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 05:34:51 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 05:34:56 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 05:34:56 App: TestLargeHandler_function got deployed
2021/12/09 05:34:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:35:01 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 05:35:06 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 05:35:11 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 05:35:16 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 05:35:16 App: TestLargeHandler_function got un-deployed
2021/12/09 05:35:16 No eventing-con process running
2021/12/09 05:35:16 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/09 05:35:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:35:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:35:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.32s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/09 05:35:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:35:39 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 05:35:39 No eventing-con process running
2021/12/09 05:35:39 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/09 05:35:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:35:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:35:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:35:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:35:57 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:35:57 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 05:35:57 No eventing-con process running
2021/12/09 05:35:57 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 05:35:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:36:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:36:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:36:10 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:36:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/09 05:36:11 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:36:16 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:36:21 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:36:26 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:36:31 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 05:36:31 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/09 05:36:31 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:36:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/09 05:36:32 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:36:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 05:36:42 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 05:36:47 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 05:36:53 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 05:36:53 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/09 05:36:53 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/09 05:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:36:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:37:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:37:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:37:13 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 05:37:13 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 05:37:13 No eventing-con process running
2021/12/09 05:37:13 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/09 05:37:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:37:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:37:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:37:32 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:37:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:37:42 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:37:47 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 05:37:47 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 05:37:47 No eventing-con process running
2021/12/09 05:37:47 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/09 05:37:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:37:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.08s)
=== RUN   TestAllowInterBucketRecursion
2021/12/09 05:38:05 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:38:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 05:38:06 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:38:11 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 05:38:16 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 05:38:21 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 05:38:26 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 05:38:26 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/09 05:38:26 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:38:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 05:38:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:38:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 05:38:37 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 05:38:42 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 05:38:47 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 05:38:47 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/09 05:38:47 TestAllowInterBucketRecursion_function2 is deployed
2021/12/09 05:38:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:38:53 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:38:58 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:39:03 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:39:08 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 05:39:08 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/09 05:39:08 No eventing-con process running
2021/12/09 05:39:08 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/09 05:39:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:39:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:39:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:39:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:39:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:39:27 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 05:39:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 05:39:37 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 05:39:42 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 05:39:42 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/09 05:39:42 No eventing-con process running
2021/12/09 05:39:42 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/09 05:39:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:39:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:39:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:39:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.19s)
=== RUN   TestInterBucketRecursion2
2021/12/09 05:40:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:40:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:40:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:40:24 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:40:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 05:40:30 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 05:40:36 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 05:40:41 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 05:40:46 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 05:40:46 App: TestInterBucketRecursion2_function1 got deployed
2021/12/09 05:40:46 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:40:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 05:40:52 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 05:40:57 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 05:41:02 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 05:41:07 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 05:41:07 App: TestInterBucketRecursion2_function2 got deployed
2021/12/09 05:41:07 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:41:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 05:41:13 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 05:41:18 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 05:41:23 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 05:41:28 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 05:41:28 App: TestInterBucketRecursion2_function3 got deployed
2021/12/09 05:41:28 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:41:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/09 05:41:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:41:33 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 05:41:38 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 05:41:43 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 05:41:48 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 05:41:48 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/09 05:41:48 No eventing-con process running
2021/12/09 05:41:48 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/09 05:41:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:41:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:42:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:42:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:42:06 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 05:42:11 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 05:42:16 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 05:42:21 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 05:42:21 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/09 05:42:21 No eventing-con process running
2021/12/09 05:42:21 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/09 05:42:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:42:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:42:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:42:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:42:40 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 05:42:45 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 05:42:50 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 05:42:55 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 05:42:55 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/09 05:42:55 No eventing-con process running
2021/12/09 05:42:55 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/09 05:42:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:43:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:43:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:43:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:43:13 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/09 05:43:13 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/09 05:43:13 No eventing-con process running
2021/12/09 05:43:13 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/09 05:43:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:43:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:43:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (212.65s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/09 05:43:33 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 05:43:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/09 05:43:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:43:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:44:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:44:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:44:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:44:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:45:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:45:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:45:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 05:45:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:47} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/09 05:46:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:46:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 05:46:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 05:46:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/09 05:46:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:46:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 05:46:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 05:46:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/09 05:46:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:46:27 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/09 05:46:27 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/09 05:46:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:46:27 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/09 05:46:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:46:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:46:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:46:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:46:45 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:46:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 05:46:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 05:46:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 05:47:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 05:47:06 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 05:47:06 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 5f4908465ee0d5807728bfbde7abcbfb
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: fadc418947a40bccbf1c26f2e6b60e8d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 3f5f36afcbd7e116a4f34075d21d037d
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/09 05:47:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:47:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 05:47:11 Rebalance progress: 29.16666666666666
2021/12/09 05:47:16 Rebalance progress: 29.16666666666666
2021/12/09 05:47:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 5f4908465ee0d5807728bfbde7abcbfb
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: fadc418947a40bccbf1c26f2e6b60e8d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 3f5f36afcbd7e116a4f34075d21d037d
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/09 05:47:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:47:21 handler_test.go:1674: Deleting source bucket: default
2021/12/09 05:47:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 05:47:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 05:47:28 Rebalance progress: 29.16666666666666
2021/12/09 05:47:33 Rebalance progress: 29.16666666666666
2021/12/09 05:47:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 120
	vb list dump: [4, 8, 32, 34, 37, 41, 49, 52, 87, 101-102, 105, 113, 118, 129, 131, 157, 164-165, 169, 174, 205, 217, 222, 224-225, 234, 240, 270, 273, 281, 291, 298-299, 307, 309-310, 313, 339, 367, 369, 379, 386, 390, 392, 414, 418, 427, 433, 435, 443, 450, 457, 460, 466-469, 471, 476, 500, 502, 504, 510-511, 592, 595, 619, 623, 633, 639-641, 645, 652, 658, 674, 685, 691, 697, 705, 708, 712, 714, 727, 731, 736, 740, 742, 751, 753, 775, 789, 795, 801, 806, 818, 829, 848, 853, 860-861, 871, 875, 880, 892-893, 910, 923, 926-927, 954-955, 972, 975, 988, 1008, 1010-1011, 1017]
	stream status: running
	len: 904
	vb list dump: [0-3, 5-7, 9-31, 33, 35-36, 38-40, 42-48, 50-51, 53-86, 88-100, 103-104, 106-112, 114-117, 119-128, 130, 132-156, 158-163, 166-168, 170-173, 175-204, 206-216, 218-221, 223, 226-233, 235-239, 241-269, 271-272, 274-280, 282-290, 292-297, 300-306, 308, 311-312, 314-338, 340-366, 368, 370-378, 380-385, 387-389, 391, 393-413, 415-417, 419-426, 428-432, 434, 436-442, 444-449, 451-456, 458-459, 461-465, 470, 472-475, 477-499, 501, 503, 505-509, 512-591, 593-594, 596-618, 620-622, 624-632, 634-638, 642-644, 646-651, 653-657, 659-673, 675-684, 686-690, 692-696, 698-704, 706-707, 709-711, 713, 715-726, 728-730, 732-735, 737-739, 741, 743-750, 752, 754-774, 776-788, 790-794, 796-800, 802-805, 807-817, 819-828, 830-847, 849-852, 854-859, 862-870, 872-874, 876-879, 881-891, 894-909, 911-922, 924-925, 928-953, 956-971, 973-974, 976-987, 989-1007, 1009, 1012-1016, 1018-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: fadc418947a40bccbf1c26f2e6b60e8d
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-922, 924-925, 928-953, 956-971, 973-974, 976-987, 989-1007, 1009, 1012-1016, 1018-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 97
	v: [683-684, 686-690, 692-696, 698-704, 706-707, 709-711, 713, 715-726, 728-730, 732-735, 737-739, 741, 743-750, 752, 754-774, 776-788, 790-794, 796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [797-800, 802-805, 807-817, 819-828, 830-847, 849-852, 854-859, 862-870, 872-874, 876-879, 881-891, 894-909]
Producer node: 127.0.0.1:9303	Node UUID: 3f5f36afcbd7e116a4f34075d21d037d
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-366, 368, 370-378, 380-385, 387-389, 391, 393-413, 415-417, 419-426, 428-432, 434, 436-442, 444-449, 451-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 458-459, 461-465, 470, 472-475, 477-499, 501, 503, 505-509, 512-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-591, 593-594, 596-618, 620-622, 624-632, 634-638, 642-644, 646-651, 653-657, 659-673, 675-682]
Producer node: 127.0.0.1:9302	Node UUID: 5f4908465ee0d5807728bfbde7abcbfb
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-233, 235-239, 241-269, 271-272, 274-280, 282-290, 292-297, 300-306, 308, 311-312, 314-338, 340-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-3, 5-7, 9-31, 33, 35-36, 38-40, 42-48, 50-51, 53-86, 88-100, 103-104, 106-112]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-117, 119-128, 130, 132-156, 158-163, 166-168, 170-173, 175-204, 206-216, 218-221, 223, 226-227]


2021/12/09 05:47:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:47:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/09 05:47:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/09 05:47:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/09 05:47:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/09 05:47:49 eventingSetup.go:46: No eventing-con process running
2021/12/09 05:47:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/09 05:47:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:47:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:47:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:48:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:48:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 05:48:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 05:48:07 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (282.27s)
=== RUN   TestN1QLRecursion
2021/12/09 05:48:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 05:48:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 05:48:11 Removing node(s): [] from the cluster
2021/12/09 05:48:16 Rebalance progress: 100
2021/12/09 05:48:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:48:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:48:33 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:48:38 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:48:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 05:48:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:48:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 05:48:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 05:48:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 05:49:00 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 05:49:00 App: TestN1QLRecursion_function1 got deployed
2021/12/09 05:49:00 TestN1QLRecursion_function1 is deployed
2021/12/09 05:49:00 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:49:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 05:49:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:49:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 05:49:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 05:49:16 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 05:49:21 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 05:49:21 App: TestN1QLRecursion_function3 got deployed
2021/12/09 05:49:21 TestN1QLRecursion_function3 is deployed
2021/12/09 05:49:21 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:49:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/09 05:49:21 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/09 05:49:21 Success: TestN1QLRecursion_function2 is not deployed
2021/12/09 05:49:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:49:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 05:49:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 05:49:36 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 05:49:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 05:49:41 App: TestN1QLRecursion_function1 got un-deployed
2021/12/09 05:49:41 No eventing-con process running
2021/12/09 05:49:41 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/09 05:49:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:49:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:49:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:49:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:50:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 05:50:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 05:50:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 05:50:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 05:50:15 App: TestN1QLRecursion_function3 got un-deployed
2021/12/09 05:50:15 No eventing-con process running
2021/12/09 05:50:15 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/09 05:50:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:50:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:50:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:50:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:50:30 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 05:50:35 Rebalance progress: 100
--- PASS: TestN1QLRecursion (144.88s)
=== RUN   TestN1QLAllowRecursion
2021/12/09 05:50:35 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 05:50:35 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 05:50:35 Removing node(s): [] from the cluster
2021/12/09 05:50:40 Rebalance progress: 100
2021/12/09 05:50:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:50:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:50:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:51:03 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:51:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 05:51:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:51:09 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 05:51:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 05:51:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 05:51:24 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 05:51:24 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/09 05:51:24 TestN1QLAllowRecursion_function1 is deployed
2021/12/09 05:51:24 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:51:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 05:51:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:51:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 05:51:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 05:51:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 05:51:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 05:51:46 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/09 05:51:46 TestN1QLAllowRecursion_function3 is deployed
2021/12/09 05:51:46 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:51:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 05:51:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/09 05:51:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 05:51:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 05:52:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 05:52:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 05:52:07 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/09 05:52:07 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/09 05:52:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:52:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 05:52:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 05:52:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 05:52:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 05:52:27 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/09 05:52:27 No eventing-con process running
2021/12/09 05:52:27 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/09 05:52:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:52:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:52:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:52:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:52:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:52:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 05:52:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 05:52:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 05:53:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 05:53:01 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/09 05:53:01 No eventing-con process running
2021/12/09 05:53:01 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/09 05:53:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:53:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:53:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:53:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:53:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:53:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 05:53:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 05:53:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 05:53:35 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 05:53:35 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/09 05:53:35 No eventing-con process running
2021/12/09 05:53:35 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/09 05:53:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:53:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:53:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:53:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:53:50 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 05:53:55 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.02s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/09 05:53:55 Sleeping for some time
2021/12/09 05:54:05 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 05:54:05 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/09 05:54:05 No eventing-con process running
2021/12/09 05:54:05 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/09 05:54:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:54:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:54:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:54:19 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:54:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 05:54:20 Starting bucket ops to source bucket
2021/12/09 05:55:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:55:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 05:55:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 05:55:14 Starting bucket ops to source bucket
2021/12/09 05:55:15 Starting bucket ops to source bucket
2021/12/09 05:56:21 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/09 05:56:21 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/09 05:56:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:56:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:56:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 05:56:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 05:56:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 05:56:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 05:56:41 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/09 05:56:41 No eventing-con process running
2021/12/09 05:56:41 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/09 05:56:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:56:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 05:56:58 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/09 05:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:57:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:57:10 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/09 05:57:15 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/09 05:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.69s)
=== RUN   TestTimerOverWriteSameReference
2021/12/09 05:57:20 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 05:57:20 App: TestTimerOverWriteSameReference got un-deployed
2021/12/09 05:57:20 No eventing-con process running
2021/12/09 05:57:20 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/09 05:57:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:57:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:57:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:57:33 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/09 05:57:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/09 05:57:39 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 05:57:44 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 05:57:49 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 05:57:54 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 05:57:54 App: TestTimerOverWriteSameReference got deployed
2021/12/09 05:58:04 Starting bucket ops to source bucket
2021/12/09 05:59:06 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/09 05:59:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 05:59:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 05:59:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 05:59:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 05:59:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 05:59:36 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 05:59:36 App: TestTimerOverWriteSameReference got un-deployed
2021/12/09 05:59:36 No eventing-con process running
2021/12/09 05:59:36 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/09 05:59:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:59:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 05:59:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 05:59:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 05:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (159.69s)
=== RUN   TestCancelTimerBucketop
2021/12/09 05:59:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 05:59:59 App: TestCancelTimerBucketop got un-deployed
2021/12/09 05:59:59 No eventing-con process running
2021/12/09 05:59:59 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/09 06:00:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:00:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:00:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:00:13 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:00:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/09 06:00:19 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 06:00:24 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 06:00:29 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 06:00:34 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 06:00:34 App: TestCancelTimerBucketop got deployed
2021/12/09 06:00:34 Starting bucket ops to source bucket
2021/12/09 06:00:42 Starting bucket ops to source bucket
2021/12/09 06:01:44 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 06:01:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:01:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:01:49 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 06:01:54 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 06:01:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 06:02:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 06:02:04 App: TestCancelTimerBucketop got un-deployed
2021/12/09 06:02:04 No eventing-con process running
2021/12/09 06:02:04 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/09 06:02:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:02:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:02:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:02:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:02:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 06:02:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (153.12s)
=== RUN   TestJSExpiryDate
2021/12/09 06:02:33 Starting bucket ops to source bucket
2021/12/09 06:02:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 06:02:46 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/09 06:02:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/09 06:03:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:03:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:03:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:03:22 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 06:03:27 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 06:03:32 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 06:03:37 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 06:03:37 App: TestJSExpiryDate got un-deployed
2021/12/09 06:03:37 No eventing-con process running
2021/12/09 06:03:37 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/09 06:03:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:03:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:03:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:03:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.61s)
=== RUN   TestBinaryDoc
2021/12/09 06:03:56 Starting bucket ops to source bucket
2021/12/09 06:03:56 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:03:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/09 06:03:57 Starting bucket ops to source bucket
2021/12/09 06:03:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:03:58 Deleting source bucket docs now
2021/12/09 06:03:58 Starting bucket ops to source bucket
2021/12/09 06:04:35 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/09 06:04:40 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/09 06:04:45 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/09 06:04:45 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/09 06:04:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:04:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:04:50 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 06:04:55 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 06:05:00 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 06:05:05 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 06:05:05 App: TestBinaryDoc got un-deployed
2021/12/09 06:05:05 No eventing-con process running
2021/12/09 06:05:05 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/09 06:05:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:05:17 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/09 06:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:05:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:05:28 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/09 06:05:33 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/09 06:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.32s)
=== RUN   TestConstantBindings
2021/12/09 06:05:33 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:05:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/09 06:05:35 Starting bucket ops to source bucket
2021/12/09 06:05:40 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 06:05:45 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 06:05:50 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 06:05:55 Waiting for dst bucket item count to get to: 6000 curr count: 1303
2021/12/09 06:06:00 Waiting for dst bucket item count to get to: 6000 curr count: 4303
2021/12/09 06:06:00 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/09 06:06:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:06:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:06:05 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 06:06:10 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 06:06:15 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 06:06:20 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 06:06:20 App: TestConstantBindings got un-deployed
2021/12/09 06:06:21 No eventing-con process running
2021/12/09 06:06:21 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/09 06:06:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:06:32 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:06:37 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 06:06:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:06:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:06:49 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/09 06:06:54 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/09 06:06:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.71s)
=== RUN   TestCodeInsight
2021/12/09 06:06:59 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 06:06:59 App: TestCodeInsight got un-deployed
2021/12/09 06:06:59 No eventing-con process running
2021/12/09 06:06:59 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/09 06:07:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:07:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:07:13 Starting bucket ops to source bucket
2021/12/09 06:07:13 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:07:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/09 06:07:20 Waiting for app: TestCodeInsight to get deployed
2021/12/09 06:07:25 Waiting for app: TestCodeInsight to get deployed
2021/12/09 06:07:30 Waiting for app: TestCodeInsight to get deployed
2021/12/09 06:07:35 Waiting for app: TestCodeInsight to get deployed
2021/12/09 06:07:35 App: TestCodeInsight got deployed
2021/12/09 06:07:40 Waiting for dst bucket item count to get to: 1000 curr count: 825
2021/12/09 06:07:40 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 06:07:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:07:45 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 06:07:50 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 06:07:55 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 06:08:00 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 06:08:00 App: TestCodeInsight got un-deployed
2021/12/09 06:08:00 No eventing-con process running
2021/12/09 06:08:00 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/09 06:08:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:08:12 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:08:17 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:08:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:08:28 Waiting for dst bucket item count to get to: 0 curr count: 824
2021/12/09 06:08:33 Waiting for dst bucket item count to get to: 0 curr count: 324
2021/12/09 06:08:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.24s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12784.940s

Functional tests took 3 hours, 33 minutes and 10 seconds.

N1ql tests

2021/12/09 06:08:42 Started n1ql
2021/12/09 06:08:53 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 06:08:53 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 06:08:53 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 06:08:53 Removing node(s): [] from the cluster
2021/12/09 06:08:58 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/09 06:09:03 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/09 06:09:03 App: TestRecursiveMutationN1QL got un-deployed
2021/12/09 06:09:03 No eventing-con process running
2021/12/09 06:09:03 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/09 06:09:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:09:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:09:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:09:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:09:17 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:09:17 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.63s)
=== RUN   TestFlexReset1
2021/12/09 06:09:22 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 06:09:22 App: TestFlexReset1 got un-deployed
2021/12/09 06:09:22 No eventing-con process running
2021/12/09 06:09:22 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/09 06:09:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:09:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:09:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:09:36 Starting bucket ops to source bucket
2021/12/09 06:09:36 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:09:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/09 06:09:42 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 06:09:47 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 06:09:52 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 06:09:57 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 06:09:57 App: TestFlexReset1 got deployed
2021/12/09 06:10:02 Waiting for dst bucket item count to get to: 1000 curr count: 37
2021/12/09 06:10:02 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 06:10:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:10:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:10:08 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 06:10:13 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 06:10:18 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 06:10:23 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 06:10:23 App: TestFlexReset1 got un-deployed
2021/12/09 06:10:23 No eventing-con process running
2021/12/09 06:10:23 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/09 06:10:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:10:34 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:10:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:10:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:10:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.13s)
=== RUN   TestFlexReset2
2021/12/09 06:10:51 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 06:10:51 App: TestFlexReset2 got un-deployed
2021/12/09 06:10:51 No eventing-con process running
2021/12/09 06:10:51 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/09 06:10:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:10:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:11:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:11:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:11:05 Starting bucket ops to source bucket
2021/12/09 06:11:05 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:11:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/09 06:11:12 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 06:11:17 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 06:11:22 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 06:11:27 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 06:11:27 App: TestFlexReset2 got deployed
2021/12/09 06:11:32 Waiting for dst bucket item count to get to: 1000 curr count: 99
2021/12/09 06:11:32 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 06:11:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:11:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:11:37 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 06:11:42 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 06:11:47 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 06:11:52 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 06:11:52 App: TestFlexReset2 got un-deployed
2021/12/09 06:11:52 No eventing-con process running
2021/12/09 06:11:52 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/09 06:11:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:12:04 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 06:12:09 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 06:12:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:12:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:12:20 Waiting for dst bucket item count to get to: 0 curr count: 785
2021/12/09 06:12:25 Waiting for dst bucket item count to get to: 0 curr count: 785
2021/12/09 06:12:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.18s)
=== RUN   TestN1QLLabelledBreak
2021/12/09 06:12:30 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 06:12:30 App: TestN1QLLabelledBreak got un-deployed
2021/12/09 06:12:30 No eventing-con process running
2021/12/09 06:12:30 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/09 06:12:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:12:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:12:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:12:44 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:12:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/09 06:12:50 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 06:12:55 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 06:13:00 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 06:13:05 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 06:13:05 App: TestN1QLLabelledBreak got deployed
2021/12/09 06:13:05 Starting bucket ops to source bucket
2021/12/09 06:13:14 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 06:13:19 Waiting for dst bucket item count to get to: 10000 curr count: 397
2021/12/09 06:13:24 Waiting for dst bucket item count to get to: 10000 curr count: 397
2021/12/09 06:13:29 Waiting for dst bucket item count to get to: 10000 curr count: 4154
2021/12/09 06:13:34 Waiting for dst bucket item count to get to: 10000 curr count: 4154
2021/12/09 06:13:39 Waiting for dst bucket item count to get to: 10000 curr count: 8247
2021/12/09 06:13:44 Waiting for dst bucket item count to get to: 10000 curr count: 8247
2021/12/09 06:13:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 06:13:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:13:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:13:50 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 06:13:55 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 06:14:00 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 06:14:05 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 06:14:05 App: TestN1QLLabelledBreak got un-deployed
2021/12/09 06:14:05 No eventing-con process running
2021/12/09 06:14:05 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/09 06:14:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:14:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:14:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:14:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:14:28 Waiting for dst bucket item count to get to: 0 curr count: 7865
2021/12/09 06:14:33 Waiting for dst bucket item count to get to: 0 curr count: 2865
2021/12/09 06:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (127.86s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/09 06:14:38 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 06:14:38 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/09 06:14:38 No eventing-con process running
2021/12/09 06:14:38 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/09 06:14:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:14:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:14:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:14:52 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:14:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/09 06:14:58 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 06:15:03 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 06:15:08 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 06:15:13 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 06:15:13 App: TestN1QLUnlabelledBreak got deployed
2021/12/09 06:15:13 Starting bucket ops to source bucket
2021/12/09 06:15:22 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 06:15:27 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 06:15:32 Waiting for dst bucket item count to get to: 10000 curr count: 1454
2021/12/09 06:15:37 Waiting for dst bucket item count to get to: 10000 curr count: 4290
2021/12/09 06:15:42 Waiting for dst bucket item count to get to: 10000 curr count: 7169
2021/12/09 06:15:47 Waiting for dst bucket item count to get to: 10000 curr count: 8882
2021/12/09 06:15:47 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 06:15:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:15:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:15:52 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 06:15:58 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 06:16:03 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 06:16:08 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 06:16:08 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/09 06:16:08 No eventing-con process running
2021/12/09 06:16:08 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/09 06:16:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:16:19 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 06:16:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 06:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:16:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:16:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 06:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (122.93s)
=== RUN   TestN1QLThrowStatement
2021/12/09 06:16:41 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 06:16:41 App: TestN1QLThrowStatement got un-deployed
2021/12/09 06:16:41 No eventing-con process running
2021/12/09 06:16:41 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/09 06:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:16:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:16:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:16:55 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:16:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/09 06:17:01 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 06:17:06 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 06:17:11 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 06:17:16 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 06:17:16 App: TestN1QLThrowStatement got deployed
2021/12/09 06:17:16 Starting bucket ops to source bucket
2021/12/09 06:17:25 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 06:17:30 Waiting for dst bucket item count to get to: 10000 curr count: 318
2021/12/09 06:17:35 Waiting for dst bucket item count to get to: 10000 curr count: 318
2021/12/09 06:17:40 Waiting for dst bucket item count to get to: 10000 curr count: 3831
2021/12/09 06:17:45 Waiting for dst bucket item count to get to: 10000 curr count: 3831
2021/12/09 06:17:50 Waiting for dst bucket item count to get to: 10000 curr count: 7795
2021/12/09 06:17:55 Waiting for dst bucket item count to get to: 10000 curr count: 7795
2021/12/09 06:17:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 06:17:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:17:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:18:00 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 06:18:05 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 06:18:10 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 06:18:15 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 06:18:15 App: TestN1QLThrowStatement got un-deployed
2021/12/09 06:18:15 No eventing-con process running
2021/12/09 06:18:15 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/09 06:18:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:18:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:18:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:18:39 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 06:18:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:18:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (127.93s)
=== RUN   TestN1QLNestedForLoop
2021/12/09 06:18:49 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 06:18:49 App: TestN1QLNestedForLoop got un-deployed
2021/12/09 06:18:49 No eventing-con process running
2021/12/09 06:18:49 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/09 06:18:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:18:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:19:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:19:03 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:19:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/09 06:19:09 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 06:19:14 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 06:19:19 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 06:19:24 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 06:19:24 App: TestN1QLNestedForLoop got deployed
2021/12/09 06:19:24 Starting bucket ops to source bucket
2021/12/09 06:19:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 06:19:38 Waiting for dst bucket item count to get to: 5000 curr count: 120
2021/12/09 06:19:43 Waiting for dst bucket item count to get to: 5000 curr count: 403
2021/12/09 06:19:48 Waiting for dst bucket item count to get to: 5000 curr count: 1532
2021/12/09 06:19:53 Waiting for dst bucket item count to get to: 5000 curr count: 2756
2021/12/09 06:19:58 Waiting for dst bucket item count to get to: 5000 curr count: 3963
2021/12/09 06:20:03 Waiting for dst bucket item count to get to: 5000 curr count: 4956
2021/12/09 06:20:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 06:20:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:20:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:20:08 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 06:20:13 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 06:20:18 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 06:20:23 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 06:20:23 App: TestN1QLNestedForLoop got un-deployed
2021/12/09 06:20:23 No eventing-con process running
2021/12/09 06:20:23 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/09 06:20:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:20:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:20:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:20:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:20:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.80s)
=== RUN   TestN1QLAutoClose
2021/12/09 06:20:52 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 06:20:52 App: TestN1QLAutoClose got un-deployed
2021/12/09 06:20:52 No eventing-con process running
2021/12/09 06:20:52 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/09 06:20:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:21:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:21:05 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:21:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/09 06:21:12 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 06:21:17 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 06:21:22 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 06:21:27 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 06:21:27 App: TestN1QLAutoClose got deployed
2021/12/09 06:21:27 Starting bucket ops to source bucket
2021/12/09 06:21:32 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/09 06:21:37 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/09 06:21:42 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/09 06:21:47 Waiting for dst bucket item count to get to: 500 curr count: 23
2021/12/09 06:21:52 Waiting for dst bucket item count to get to: 500 curr count: 23
2021/12/09 06:21:57 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/09 06:22:02 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/09 06:22:07 Waiting for dst bucket item count to get to: 500 curr count: 69
2021/12/09 06:22:12 Waiting for dst bucket item count to get to: 500 curr count: 69
2021/12/09 06:22:17 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/09 06:22:22 Waiting for dst bucket item count to get to: 500 curr count: 91
2021/12/09 06:22:27 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/09 06:22:32 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/09 06:22:37 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/09 06:22:42 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/09 06:22:47 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/09 06:22:52 Waiting for dst bucket item count to get to: 500 curr count: 158
2021/12/09 06:22:57 Waiting for dst bucket item count to get to: 500 curr count: 168
2021/12/09 06:23:02 Waiting for dst bucket item count to get to: 500 curr count: 179
2021/12/09 06:23:07 Waiting for dst bucket item count to get to: 500 curr count: 190
2021/12/09 06:23:12 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/09 06:23:17 Waiting for dst bucket item count to get to: 500 curr count: 213
2021/12/09 06:23:22 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/09 06:23:27 Waiting for dst bucket item count to get to: 500 curr count: 233
2021/12/09 06:23:32 Waiting for dst bucket item count to get to: 500 curr count: 245
2021/12/09 06:23:37 Waiting for dst bucket item count to get to: 500 curr count: 254
2021/12/09 06:23:42 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/09 06:23:47 Waiting for dst bucket item count to get to: 500 curr count: 274
2021/12/09 06:23:53 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/09 06:23:58 Waiting for dst bucket item count to get to: 500 curr count: 298
2021/12/09 06:24:03 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/09 06:24:08 Waiting for dst bucket item count to get to: 500 curr count: 319
2021/12/09 06:24:13 Waiting for dst bucket item count to get to: 500 curr count: 330
2021/12/09 06:24:18 Waiting for dst bucket item count to get to: 500 curr count: 350
2021/12/09 06:24:23 Waiting for dst bucket item count to get to: 500 curr count: 350
2021/12/09 06:24:28 Waiting for dst bucket item count to get to: 500 curr count: 372
2021/12/09 06:24:33 Waiting for dst bucket item count to get to: 500 curr count: 372
2021/12/09 06:24:38 Waiting for dst bucket item count to get to: 500 curr count: 394
2021/12/09 06:24:43 Waiting for dst bucket item count to get to: 500 curr count: 394
2021/12/09 06:24:48 Waiting for dst bucket item count to get to: 500 curr count: 415
2021/12/09 06:24:53 Waiting for dst bucket item count to get to: 500 curr count: 415
2021/12/09 06:24:58 Waiting for dst bucket item count to get to: 500 curr count: 437
2021/12/09 06:25:03 Waiting for dst bucket item count to get to: 500 curr count: 437
2021/12/09 06:25:08 Waiting for dst bucket item count to get to: 500 curr count: 459
2021/12/09 06:25:13 Waiting for dst bucket item count to get to: 500 curr count: 459
2021/12/09 06:25:18 Waiting for dst bucket item count to get to: 500 curr count: 482
2021/12/09 06:25:23 Waiting for dst bucket item count to get to: 500 curr count: 482
2021/12/09 06:25:28 Waiting for dst bucket item count to get to: 500 curr count: 499
2021/12/09 06:25:33 Waiting for dst bucket item count to get to: 500 curr count: 499
2021/12/09 06:25:33 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/09 06:25:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:25:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:25:38 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 06:25:43 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 06:25:48 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 06:25:53 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 06:25:53 App: TestN1QLAutoClose got un-deployed
2021/12/09 06:25:53 No eventing-con process running
2021/12/09 06:25:53 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/09 06:25:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:26:05 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 06:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:26:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:26:16 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 06:26:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.50s)
=== RUN   TestN1QLPosParams
2021/12/09 06:26:21 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 06:26:21 App: TestN1QLPosParams got un-deployed
2021/12/09 06:26:21 No eventing-con process running
2021/12/09 06:26:21 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/09 06:26:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:26:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:26:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:26:35 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:26:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/09 06:26:41 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 06:26:46 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 06:26:51 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 06:26:56 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 06:26:56 App: TestN1QLPosParams got deployed
2021/12/09 06:26:56 Starting bucket ops to source bucket
2021/12/09 06:27:06 Waiting for dst bucket item count to get to: 5000 curr count: 665
2021/12/09 06:27:11 Waiting for dst bucket item count to get to: 5000 curr count: 665
2021/12/09 06:27:11 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 06:27:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:27:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:27:16 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 06:27:21 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 06:27:26 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 06:27:31 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 06:27:31 App: TestN1QLPosParams got un-deployed
2021/12/09 06:27:31 No eventing-con process running
2021/12/09 06:27:31 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/09 06:27:33 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:27:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:27:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 06:27:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:27:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:27:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 06:28:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 06:28:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.12s)
=== RUN   TestN1QLExhaustConnPool
2021/12/09 06:28:09 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 06:28:10 App: TestN1QLExhaustConnPool got un-deployed
2021/12/09 06:28:10 No eventing-con process running
2021/12/09 06:28:10 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/09 06:28:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:28:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:28:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:28:23 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:28:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/09 06:28:29 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 06:28:34 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 06:28:39 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 06:28:44 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 06:28:44 App: TestN1QLExhaustConnPool got deployed
2021/12/09 06:28:44 Starting bucket ops to source bucket
2021/12/09 06:28:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 06:28:55 Waiting for dst bucket item count to get to: 100 curr count: 44
2021/12/09 06:29:00 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/09 06:29:00 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 06:29:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:29:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:29:05 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 06:29:10 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 06:29:15 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 06:29:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 06:29:20 App: TestN1QLExhaustConnPool got un-deployed
2021/12/09 06:29:20 No eventing-con process running
2021/12/09 06:29:20 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/09 06:29:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:29:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 06:29:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 06:29:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:29:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:29:48 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/09 06:29:53 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/09 06:29:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.93s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/09 06:29:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 06:29:58 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/09 06:29:58 No eventing-con process running
2021/12/09 06:29:58 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/09 06:30:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:30:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:30:12 Starting bucket ops to source bucket
2021/12/09 06:30:29 Starting bucket ops to source bucket
2021/12/09 06:30:47 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 15048
2021/12/09 06:30:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/09 06:30:53 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 06:30:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 06:31:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 06:31:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 06:31:08 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/09 06:31:08 Starting bucket ops to source bucket
2021/12/09 06:31:13 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:18 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:28 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 06:31:53 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/09 06:31:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:31:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:31:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 06:32:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 06:32:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 06:32:14 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 06:32:14 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/09 06:32:14 No eventing-con process running
2021/12/09 06:32:14 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/09 06:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:32:25 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/09 06:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:32:37 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/09 06:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (163.71s)
=== RUN   TestN1QLQueryConsistency
2021/12/09 06:32:42 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 06:32:42 App: n1qlquery_consistency got un-deployed
2021/12/09 06:32:42 No eventing-con process running
2021/12/09 06:32:42 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/09 06:32:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:32:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:32:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:32:56 Starting bucket ops to source bucket
2021/12/09 06:32:59 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:33:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/09 06:33:05 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 06:33:10 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 06:33:15 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 06:33:20 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 06:33:20 App: n1qlquery_consistency got deployed
2021/12/09 06:33:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 06:33:30 Waiting for dst bucket item count to get to: 5000 curr count: 4051
2021/12/09 06:33:35 Waiting for dst bucket item count to get to: 5000 curr count: 4051
2021/12/09 06:33:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 06:33:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 06:33:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:33:40 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 06:33:45 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 06:33:50 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 06:33:55 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 06:33:55 App: n1qlquery_consistency got un-deployed
2021/12/09 06:33:55 No eventing-con process running
2021/12/09 06:33:55 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/09 06:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:34:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 06:34:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:34:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 06:34:19 Waiting for dst bucket item count to get to: 0 curr count: 4604
2021/12/09 06:34:24 Waiting for dst bucket item count to get to: 0 curr count: 2104
2021/12/09 06:34:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.60s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1536.440s

N1ql tests took 0 hours, 25 minutes and 42 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/09 06:35:56 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:57040->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:57040->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/09 06:36:15 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/09 06:36:20 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 06:36:20 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/09 06:36:20 No eventing-con process running
2021/12/09 06:36:20 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/09 06:36:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:36:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:36:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:36:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:36:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:36:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:36:38 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:36:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 06:36:44 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 06:36:49 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 06:36:54 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 06:36:59 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 06:36:59 App: TestDuplicateWithEventingRebIn got deployed
2021/12/09 06:36:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 06:36:59 Starting bucket ops to source bucket
2021/12/09 06:37:14 Removing node(s): [] from the cluster
2021/12/09 06:37:19 Rebalance progress: 0
2021/12/09 06:37:24 Rebalance progress: 0
2021/12/09 06:37:29 Rebalance progress: 25
2021/12/09 06:37:34 Rebalance progress: 25
2021/12/09 06:37:39 Rebalance progress: 25
2021/12/09 06:37:44 Rebalance progress: 25
2021/12/09 06:37:49 Rebalance progress: 25
2021/12/09 06:37:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:37:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:37:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:37:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 06:37:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:37:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 06:37:59 Rebalance progress: 25
2021/12/09 06:38:04 Rebalance progress: 25
2021/12/09 06:38:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:38:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:38:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:38:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 06:38:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:38:09 Waiting for src bucket item count to get to: 200000 curr count: 97997
2021/12/09 06:38:14 Waiting for src bucket item count to get to: 200000 curr count: 114871
2021/12/09 06:38:19 Waiting for src bucket item count to get to: 200000 curr count: 114871
2021/12/09 06:38:24 Waiting for src bucket item count to get to: 200000 curr count: 131701
2021/12/09 06:38:29 Waiting for src bucket item count to get to: 200000 curr count: 131701
2021/12/09 06:38:34 Waiting for src bucket item count to get to: 200000 curr count: 148637
2021/12/09 06:38:39 Waiting for src bucket item count to get to: 200000 curr count: 148637
2021/12/09 06:38:44 Waiting for src bucket item count to get to: 200000 curr count: 165452
2021/12/09 06:38:49 Waiting for src bucket item count to get to: 200000 curr count: 165452
2021/12/09 06:38:54 Waiting for src bucket item count to get to: 200000 curr count: 182197
2021/12/09 06:38:59 Waiting for src bucket item count to get to: 200000 curr count: 182197
2021/12/09 06:39:04 Waiting for src bucket item count to get to: 200000 curr count: 199053
2021/12/09 06:39:09 Waiting for src bucket item count to get to: 200000 curr count: 199053
2021/12/09 06:39:19 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:24 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:29 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:34 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:39 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:44 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:49 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:54 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:39:59 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:40:04 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 06:40:04 Post SET ops. Expected item count: 200000 got 200001
2021/12/09 06:40:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:40:09 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 06:40:14 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 06:40:19 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 06:40:24 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 06:40:24 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/09 06:40:24 No eventing-con process running
2021/12/09 06:40:24 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/09 06:40:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:40:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:40:36 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:40:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:40:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:40:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:40:48 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/09 06:40:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (282.93s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/09 06:40:53 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 06:40:53 Removing node(s): [] from the cluster
2021/12/09 06:40:58 Rebalance progress: 100
2021/12/09 06:41:03 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 06:41:03 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/09 06:41:03 No eventing-con process running
2021/12/09 06:41:03 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/09 06:41:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:41:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:41:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:41:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:41:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:41:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:41:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:41:22 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:41:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 06:41:28 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 06:41:33 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 06:41:38 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 06:41:43 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 06:41:43 App: TestDuplicateWithEventingRebOut got deployed
2021/12/09 06:41:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 06:41:43 Starting bucket ops to source bucket
2021/12/09 06:41:48 Rebalance progress: 25
2021/12/09 06:41:53 Rebalance progress: 25
2021/12/09 06:41:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:41:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:41:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:41:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 06:41:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:41:58 Waiting for src bucket item count to get to: 200000 curr count: 10089
2021/12/09 06:42:03 Waiting for src bucket item count to get to: 200000 curr count: 26859
2021/12/09 06:42:08 Waiting for src bucket item count to get to: 200000 curr count: 26859
2021/12/09 06:42:13 Waiting for src bucket item count to get to: 200000 curr count: 43684
2021/12/09 06:42:18 Waiting for src bucket item count to get to: 200000 curr count: 43684
2021/12/09 06:42:23 Waiting for src bucket item count to get to: 200000 curr count: 60562
2021/12/09 06:42:28 Waiting for src bucket item count to get to: 200000 curr count: 60562
2021/12/09 06:42:33 Waiting for src bucket item count to get to: 200000 curr count: 60562
2021/12/09 06:42:38 Waiting for src bucket item count to get to: 200000 curr count: 77208
2021/12/09 06:42:43 Waiting for src bucket item count to get to: 200000 curr count: 77208
2021/12/09 06:42:48 Waiting for src bucket item count to get to: 200000 curr count: 93854
2021/12/09 06:42:53 Waiting for src bucket item count to get to: 200000 curr count: 93854
2021/12/09 06:42:58 Waiting for src bucket item count to get to: 200000 curr count: 110744
2021/12/09 06:43:03 Waiting for src bucket item count to get to: 200000 curr count: 110744
2021/12/09 06:43:08 Waiting for src bucket item count to get to: 200000 curr count: 127765
2021/12/09 06:43:13 Waiting for src bucket item count to get to: 200000 curr count: 127765
2021/12/09 06:43:18 Waiting for src bucket item count to get to: 200000 curr count: 144452
2021/12/09 06:43:23 Waiting for src bucket item count to get to: 200000 curr count: 144452
2021/12/09 06:43:28 Waiting for src bucket item count to get to: 200000 curr count: 161323
2021/12/09 06:43:33 Waiting for src bucket item count to get to: 200000 curr count: 161323
2021/12/09 06:43:38 Waiting for src bucket item count to get to: 200000 curr count: 177960
2021/12/09 06:43:43 Waiting for src bucket item count to get to: 200000 curr count: 177960
2021/12/09 06:43:48 Waiting for src bucket item count to get to: 200000 curr count: 194625
2021/12/09 06:43:53 Waiting for src bucket item count to get to: 200000 curr count: 194625
2021/12/09 06:43:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 06:43:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 06:43:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:44:03 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 06:44:08 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 06:44:13 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 06:44:18 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 06:44:18 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/09 06:44:18 No eventing-con process running
2021/12/09 06:44:18 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/09 06:44:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:44:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:44:30 Waiting for dst bucket item count to get to: 0 curr count: 23650
2021/12/09 06:44:35 Waiting for dst bucket item count to get to: 0 curr count: 23650
2021/12/09 06:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:44:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:44:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:44:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.92s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/09 06:44:57 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 06:44:57 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/09 06:44:57 No eventing-con process running
2021/12/09 06:44:57 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/09 06:44:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:45:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:45:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:45:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:45:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:45:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:45:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:45:15 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:45:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 06:45:21 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 06:45:26 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 06:45:31 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 06:45:36 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 06:45:36 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/09 06:45:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 06:45:36 Starting bucket ops to source bucket
2021/12/09 06:45:37 Removing node(s): [] from the cluster
2021/12/09 06:45:42 Rebalance progress: 25
2021/12/09 06:45:47 Rebalance progress: 25
2021/12/09 06:45:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:45:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 06:45:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:45:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 06:45:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 06:46:05 Rebalance progress: 16.66666666666667
2021/12/09 06:46:10 Rebalance progress: 16.66666666666667
2021/12/09 06:46:15 Rebalance progress: 16.66666666666667
2021/12/09 06:46:20 Rebalance progress: 16.66666666666667
2021/12/09 06:46:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:46:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:46:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 06:46:30 Rebalance progress: 16.66666666666667
2021/12/09 06:46:35 Rebalance progress: 16.66666666666667
2021/12/09 06:46:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:46:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:46:40 Waiting for src bucket item count to get to: 200000 curr count: 79831
2021/12/09 06:46:45 Waiting for src bucket item count to get to: 200000 curr count: 96459
2021/12/09 06:46:50 Waiting for src bucket item count to get to: 200000 curr count: 96459
2021/12/09 06:46:55 Waiting for src bucket item count to get to: 200000 curr count: 113181
2021/12/09 06:47:00 Waiting for src bucket item count to get to: 200000 curr count: 113181
2021/12/09 06:47:05 Waiting for src bucket item count to get to: 200000 curr count: 130125
2021/12/09 06:47:10 Waiting for src bucket item count to get to: 200000 curr count: 130125
2021/12/09 06:47:15 Waiting for src bucket item count to get to: 200000 curr count: 146914
2021/12/09 06:47:20 Waiting for src bucket item count to get to: 200000 curr count: 146914
2021/12/09 06:47:25 Waiting for src bucket item count to get to: 200000 curr count: 163391
2021/12/09 06:47:30 Waiting for src bucket item count to get to: 200000 curr count: 163391
2021/12/09 06:47:35 Waiting for src bucket item count to get to: 200000 curr count: 179335
2021/12/09 06:47:40 Waiting for src bucket item count to get to: 200000 curr count: 179335
2021/12/09 06:47:45 Waiting for src bucket item count to get to: 200000 curr count: 195763
2021/12/09 06:47:50 Waiting for src bucket item count to get to: 200000 curr count: 195763
2021/12/09 06:47:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 06:47:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 06:47:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:48:00 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 06:48:05 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 06:48:10 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 06:48:15 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 06:48:15 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/09 06:48:15 No eventing-con process running
2021/12/09 06:48:15 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/09 06:48:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:48:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:48:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:48:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:48:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:48:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:48:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.54s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/09 06:48:48 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 06:48:48 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/09 06:48:48 No eventing-con process running
2021/12/09 06:48:48 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/09 06:48:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:48:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:48:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:48:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:49:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:49:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:49:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:49:07 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:49:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 06:49:13 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 06:49:18 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 06:49:23 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 06:49:28 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 06:49:28 App: TestDuplicateWithKVRebIn got deployed
2021/12/09 06:49:28 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 06:49:28 Starting bucket ops to source bucket
2021/12/09 06:49:28 Removing node(s): [] from the cluster
2021/12/09 06:49:33 Rebalance progress: 16.66666666666667
2021/12/09 06:49:38 Rebalance progress: 16.66666666666667
2021/12/09 06:49:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:49:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:49:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 06:49:48 Rebalance progress: 16.66666666666667
2021/12/09 06:49:53 Rebalance progress: 16.66666666666667
2021/12/09 06:49:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:49:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:49:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:49:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:49:58 Waiting for src bucket item count to get to: 200000 curr count: 33680
2021/12/09 06:50:03 Waiting for src bucket item count to get to: 200000 curr count: 50345
2021/12/09 06:50:08 Waiting for src bucket item count to get to: 200000 curr count: 50345
2021/12/09 06:50:13 Waiting for src bucket item count to get to: 200000 curr count: 67069
2021/12/09 06:50:18 Waiting for src bucket item count to get to: 200000 curr count: 67069
2021/12/09 06:50:23 Waiting for src bucket item count to get to: 200000 curr count: 82895
2021/12/09 06:50:28 Waiting for src bucket item count to get to: 200000 curr count: 82895
2021/12/09 06:50:33 Waiting for src bucket item count to get to: 200000 curr count: 82895
2021/12/09 06:50:38 Waiting for src bucket item count to get to: 200000 curr count: 99385
2021/12/09 06:50:43 Waiting for src bucket item count to get to: 200000 curr count: 99385
2021/12/09 06:50:48 Waiting for src bucket item count to get to: 200000 curr count: 116027
2021/12/09 06:50:53 Waiting for src bucket item count to get to: 200000 curr count: 116027
2021/12/09 06:50:58 Waiting for src bucket item count to get to: 200000 curr count: 132617
2021/12/09 06:51:03 Waiting for src bucket item count to get to: 200000 curr count: 132617
2021/12/09 06:51:08 Waiting for src bucket item count to get to: 200000 curr count: 149423
2021/12/09 06:51:13 Waiting for src bucket item count to get to: 200000 curr count: 149423
2021/12/09 06:51:18 Waiting for src bucket item count to get to: 200000 curr count: 165807
2021/12/09 06:51:23 Waiting for src bucket item count to get to: 200000 curr count: 165807
2021/12/09 06:51:28 Waiting for src bucket item count to get to: 200000 curr count: 181671
2021/12/09 06:51:33 Waiting for src bucket item count to get to: 200000 curr count: 181671
2021/12/09 06:51:38 Waiting for src bucket item count to get to: 200000 curr count: 198265
2021/12/09 06:51:43 Waiting for src bucket item count to get to: 200000 curr count: 198265
2021/12/09 06:51:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 06:51:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 06:51:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:51:54 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 06:51:59 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 06:52:04 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 06:52:09 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 06:52:09 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/09 06:52:09 No eventing-con process running
2021/12/09 06:52:09 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/09 06:52:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:52:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:52:20 Waiting for dst bucket item count to get to: 0 curr count: 80887
2021/12/09 06:52:25 Waiting for dst bucket item count to get to: 0 curr count: 80887
2021/12/09 06:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:52:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:52:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:52:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:52:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (238.73s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/09 06:52:42 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 06:52:43 Removing node(s): [] from the cluster
2021/12/09 06:52:48 Rebalance progress: 100
2021/12/09 06:52:53 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 06:52:53 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/09 06:52:53 No eventing-con process running
2021/12/09 06:52:53 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/09 06:52:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:52:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:53:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:53:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:53:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:53:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:53:11 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:53:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 06:53:17 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 06:53:22 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 06:53:27 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 06:53:32 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 06:53:32 App: TestDuplicateWithKVRebOut got deployed
2021/12/09 06:53:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 06:53:32 Starting bucket ops to source bucket
2021/12/09 06:53:37 Rebalance progress: 16.66666666666667
2021/12/09 06:53:42 Rebalance progress: 16.66666666666667
2021/12/09 06:53:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:53:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:53:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:53:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:53:47 Waiting for src bucket item count to get to: 200000 curr count: 11010
2021/12/09 06:53:52 Waiting for src bucket item count to get to: 200000 curr count: 27385
2021/12/09 06:53:57 Waiting for src bucket item count to get to: 200000 curr count: 27385
2021/12/09 06:54:02 Waiting for src bucket item count to get to: 200000 curr count: 43769
2021/12/09 06:54:07 Waiting for src bucket item count to get to: 200000 curr count: 43769
2021/12/09 06:54:12 Waiting for src bucket item count to get to: 200000 curr count: 59958
2021/12/09 06:54:17 Waiting for src bucket item count to get to: 200000 curr count: 59958
2021/12/09 06:54:22 Waiting for src bucket item count to get to: 200000 curr count: 59958
2021/12/09 06:54:27 Waiting for src bucket item count to get to: 200000 curr count: 75474
2021/12/09 06:54:32 Waiting for src bucket item count to get to: 200000 curr count: 75474
2021/12/09 06:54:37 Waiting for src bucket item count to get to: 200000 curr count: 91633
2021/12/09 06:54:42 Waiting for src bucket item count to get to: 200000 curr count: 91633
2021/12/09 06:54:47 Waiting for src bucket item count to get to: 200000 curr count: 107881
2021/12/09 06:54:52 Waiting for src bucket item count to get to: 200000 curr count: 107881
2021/12/09 06:54:57 Waiting for src bucket item count to get to: 200000 curr count: 124365
2021/12/09 06:55:02 Waiting for src bucket item count to get to: 200000 curr count: 124365
2021/12/09 06:55:07 Waiting for src bucket item count to get to: 200000 curr count: 140705
2021/12/09 06:55:12 Waiting for src bucket item count to get to: 200000 curr count: 140705
2021/12/09 06:55:17 Waiting for src bucket item count to get to: 200000 curr count: 156994
2021/12/09 06:55:22 Waiting for src bucket item count to get to: 200000 curr count: 156994
2021/12/09 06:55:27 Waiting for src bucket item count to get to: 200000 curr count: 172492
2021/12/09 06:55:32 Waiting for src bucket item count to get to: 200000 curr count: 172492
2021/12/09 06:55:37 Waiting for src bucket item count to get to: 200000 curr count: 188497
2021/12/09 06:55:42 Waiting for src bucket item count to get to: 200000 curr count: 188497
2021/12/09 06:55:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 06:55:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 06:55:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:55:53 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 06:55:58 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 06:56:03 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 06:56:08 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 06:56:08 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/09 06:56:08 No eventing-con process running
2021/12/09 06:56:08 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/09 06:56:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:56:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:56:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:56:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:56:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:56:26 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.02s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/09 06:56:36 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 06:56:36 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/09 06:56:36 No eventing-con process running
2021/12/09 06:56:36 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/09 06:56:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:56:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:56:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 06:56:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:56:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:56:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:56:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:56:55 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 06:56:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 06:57:01 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 06:57:06 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 06:57:11 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 06:57:16 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 06:57:16 App: TestDuplicateWithKVSwapReb got deployed
2021/12/09 06:57:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 06:57:16 Starting bucket ops to source bucket
2021/12/09 06:57:16 Removing node(s): [] from the cluster
2021/12/09 06:57:21 Rebalance progress: 16.66666666666667
2021/12/09 06:57:26 Rebalance progress: 16.66666666666667
2021/12/09 06:57:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:57:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:57:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:57:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:57:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 06:57:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 06:57:36 Rebalance progress: 16.66666666666667
2021/12/09 06:57:41 Rebalance progress: 16.66666666666667
2021/12/09 06:57:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:57:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:57:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:57:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:57:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 06:57:51 Rebalance progress: 16.66666666666667
2021/12/09 06:57:56 Rebalance progress: 16.66666666666667
2021/12/09 06:58:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:58:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:58:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 06:58:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 06:58:02 Waiting for src bucket item count to get to: 200000 curr count: 69805
2021/12/09 06:58:07 Waiting for src bucket item count to get to: 200000 curr count: 69805
2021/12/09 06:58:12 Waiting for src bucket item count to get to: 200000 curr count: 85009
2021/12/09 06:58:17 Waiting for src bucket item count to get to: 200000 curr count: 85009
2021/12/09 06:58:22 Waiting for src bucket item count to get to: 200000 curr count: 101383
2021/12/09 06:58:27 Waiting for src bucket item count to get to: 200000 curr count: 101383
2021/12/09 06:58:32 Waiting for src bucket item count to get to: 200000 curr count: 117819
2021/12/09 06:58:37 Waiting for src bucket item count to get to: 200000 curr count: 117819
2021/12/09 06:58:42 Waiting for src bucket item count to get to: 200000 curr count: 134783
2021/12/09 06:58:47 Waiting for src bucket item count to get to: 200000 curr count: 134783
2021/12/09 06:58:52 Waiting for src bucket item count to get to: 200000 curr count: 134783
2021/12/09 06:58:57 Waiting for src bucket item count to get to: 200000 curr count: 151571
2021/12/09 06:59:02 Waiting for src bucket item count to get to: 200000 curr count: 151571
2021/12/09 06:59:07 Waiting for src bucket item count to get to: 200000 curr count: 168393
2021/12/09 06:59:12 Waiting for src bucket item count to get to: 200000 curr count: 168393
2021/12/09 06:59:17 Waiting for src bucket item count to get to: 200000 curr count: 184155
2021/12/09 06:59:22 Waiting for src bucket item count to get to: 200000 curr count: 184155
2021/12/09 06:59:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 06:59:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 06:59:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 06:59:32 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 06:59:37 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 06:59:42 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 06:59:47 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 06:59:47 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/09 06:59:47 No eventing-con process running
2021/12/09 06:59:47 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/09 06:59:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 06:59:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 06:59:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 06:59:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:00:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:00:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:00:10 Waiting for dst bucket item count to get to: 0 curr count: 70714
2021/12/09 07:00:15 Waiting for dst bucket item count to get to: 0 curr count: 70714
2021/12/09 07:00:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.17s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/09 07:00:25 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:00:25 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 07:00:25 No eventing-con process running
2021/12/09 07:00:25 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 07:00:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:00:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:00:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:00:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:00:44 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:00:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 07:00:50 Starting bucket ops to source bucket
2021/12/09 07:00:55 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:01:00 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:01:05 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:01:05 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:01:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 07:01:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:01:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:01:10 Rebalance progress: 16.66666666666667
2021/12/09 07:01:15 Rebalance progress: 16.66666666666667
2021/12/09 07:01:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:01:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:01:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:01:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 07:01:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:01:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:01:25 Rebalance progress: 16.66666666666667
2021/12/09 07:01:30 Rebalance progress: 16.66666666666667
2021/12/09 07:01:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:01:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:01:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:01:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 07:01:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 07:01:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:01:46 Rebalance progress: 12.5
2021/12/09 07:01:51 Rebalance progress: 12.5
2021/12/09 07:01:56 Rebalance progress: 12.5
2021/12/09 07:02:01 Rebalance progress: 98.6
2021/12/09 07:02:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:02:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:02:11 Rebalance progress: 12.5
2021/12/09 07:02:16 Rebalance progress: 12.5
2021/12/09 07:02:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:02:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:02:26 Rebalance progress: 12.5
2021/12/09 07:02:31 Rebalance progress: 12.5
2021/12/09 07:02:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:02:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 07:02:41 Rebalance progress: 12.5
2021/12/09 07:02:46 Rebalance progress: 12.5
2021/12/09 07:02:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:02:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:02:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:02:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:03:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:03:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:03:11 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:03:11 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 07:03:11 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:03:11 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 07:03:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:03:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:03:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12113
2021/12/09 07:03:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12113
2021/12/09 07:03:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:03:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:03:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:03:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.56s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/09 07:03:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 07:03:45 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/09 07:03:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:03:45 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/09 07:03:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:03:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:03:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:03:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:03:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:03:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:04:03 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:04:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/09 07:04:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:04:14 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 07:04:19 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 07:04:24 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 07:04:24 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:04:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:04:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:04:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 07:04:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:04:30 Rebalance progress: 12.5
2021/12/09 07:04:35 Rebalance progress: 12.5
2021/12/09 07:04:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:04:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 07:04:46 Rebalance progress: 12.5
2021/12/09 07:04:51 Rebalance progress: 12.5
2021/12/09 07:04:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:04:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:04:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:05:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 07:05:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 07:05:11 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 07:05:16 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 07:05:16 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/09 07:05:16 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:05:16 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/09 07:05:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:05:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:05:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4590
2021/12/09 07:05:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:05:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:05:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:05:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4590
2021/12/09 07:05:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (124.39s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/09 07:05:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 07:05:49 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 07:05:49 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:05:49 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/09 07:05:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:05:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:05:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:05:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:06:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:06:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:06:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:06:08 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:06:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 07:06:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:06:19 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 07:06:24 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 07:06:29 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 07:06:29 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:06:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:06:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:06:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:06:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:06:34 Rebalance progress: 12.5
2021/12/09 07:06:39 Rebalance progress: 12.5
2021/12/09 07:06:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:06:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:06:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:06:50 Rebalance progress: 12.5
2021/12/09 07:06:55 Rebalance progress: 12.5
2021/12/09 07:07:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:07:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 07:07:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:07:06 Rebalance progress: 12.5
2021/12/09 07:07:11 Rebalance progress: 12.5
2021/12/09 07:07:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:07:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:07:21 Rebalance progress: 12.5
2021/12/09 07:07:26 Rebalance progress: 12.5
2021/12/09 07:07:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:07:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:07:36 Rebalance progress: 12.5
2021/12/09 07:07:41 Rebalance progress: 12.5
2021/12/09 07:07:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:07:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:07:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 07:07:51 Rebalance progress: 12.5
2021/12/09 07:07:56 Rebalance progress: 12.5
2021/12/09 07:08:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:08:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:08:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:08:01 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:08:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 07:08:11 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 07:08:16 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 07:08:21 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 07:08:21 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 07:08:21 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:08:21 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/09 07:08:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:08:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:08:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10715
2021/12/09 07:08:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10715
2021/12/09 07:08:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:08:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:08:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:08:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (185.49s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/09 07:08:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 07:08:55 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/09 07:08:55 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:08:55 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/09 07:08:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:09:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:09:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:09:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:09:08 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:09:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 07:09:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:09:19 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 07:09:24 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 07:09:29 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 07:09:29 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:09:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:09:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:09:35 Rebalance progress: 12.5
2021/12/09 07:09:40 Rebalance progress: 12.5
2021/12/09 07:09:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:09:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:09:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:09:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:09:51 Rebalance progress: 12.5
2021/12/09 07:09:56 Rebalance progress: 12.5
2021/12/09 07:10:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:10:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:10:06 Rebalance progress: 12.5
2021/12/09 07:10:11 Rebalance progress: 12.5
2021/12/09 07:10:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:10:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:10:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 07:10:26 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 07:10:31 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 07:10:36 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 07:10:36 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/09 07:10:36 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:10:36 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/09 07:10:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:10:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:10:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6151
2021/12/09 07:10:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:10:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:10:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:10:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.84s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/09 07:11:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:11:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:11:05 Rebalance progress: 100
2021/12/09 07:11:10 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 07:11:10 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/09 07:11:10 No eventing-con process running
2021/12/09 07:11:10 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/09 07:11:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:11:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:11:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:11:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:11:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:11:24 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:11:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 07:11:30 Starting bucket ops to source bucket
2021/12/09 07:11:35 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 07:11:40 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 07:11:45 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 07:11:45 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:11:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:11:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:11:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:11:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:11:50 Rebalance progress: 12.5
2021/12/09 07:11:55 Rebalance progress: 12.5
2021/12/09 07:12:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:12:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:12:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:12:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 07:12:00 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 07:12:05 Rebalance progress: 12.5
2021/12/09 07:12:10 Rebalance progress: 12.5
2021/12/09 07:12:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:12:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:12:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:12:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:12:20 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 07:12:25 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 07:12:30 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 07:12:35 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 07:12:35 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/09 07:12:35 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:12:35 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/09 07:12:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:12:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:12:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4550
2021/12/09 07:12:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:12:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:12:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:12:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4550
2021/12/09 07:12:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (124.31s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/09 07:13:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:13:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:13:09 Rebalance progress: 100
2021/12/09 07:13:14 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 07:13:14 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/09 07:13:14 No eventing-con process running
2021/12/09 07:13:14 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/09 07:13:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:13:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:13:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:13:28 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:13:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 07:13:34 Starting bucket ops to source bucket
2021/12/09 07:13:39 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 07:13:44 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 07:13:49 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 07:13:49 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:13:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:13:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:13:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:13:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:13:55 Rebalance progress: 12.5
2021/12/09 07:14:00 Rebalance progress: 12.5
2021/12/09 07:14:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:14:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:14:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:14:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 07:14:05 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 07:14:10 Rebalance progress: 12.5
2021/12/09 07:14:15 Rebalance progress: 12.5
2021/12/09 07:14:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:14:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:14:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:14:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:14:20 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 07:14:25 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 07:14:30 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 07:14:35 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 07:14:40 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 07:14:40 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/09 07:14:40 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:14:40 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/09 07:14:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:14:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:14:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38474
2021/12/09 07:14:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38474
2021/12/09 07:14:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:14:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:15:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:15:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38474
2021/12/09 07:15:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.79s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/09 07:15:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 07:15:19 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/09 07:15:19 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:15:19 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/09 07:15:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:15:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:15:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:15:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:15:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:15:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:15:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:15:37 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:15:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:15:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:15:38 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:15:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 07:15:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:15:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 07:15:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 07:16:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 07:16:00 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/09 07:16:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 07:16:10 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 07:16:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 07:16:15 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:16:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:16:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:16:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:16:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:16:20 Rebalance progress: 12.5
2021/12/09 07:16:25 Rebalance progress: 12.5
2021/12/09 07:16:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:16:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:16:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:16:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:16:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:16:35 Rebalance progress: 12.5
2021/12/09 07:16:40 Rebalance progress: 12.5
2021/12/09 07:16:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:16:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:16:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:16:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 07:16:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:16:51 Rebalance progress: 12.5
2021/12/09 07:16:56 Rebalance progress: 12.5
2021/12/09 07:17:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:17:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:17:06 Rebalance progress: 12.5
2021/12/09 07:17:11 Rebalance progress: 12.5
2021/12/09 07:17:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:17:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:17:21 Rebalance progress: 12.5
2021/12/09 07:17:26 Rebalance progress: 12.5
2021/12/09 07:17:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:17:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 07:17:36 Rebalance progress: 12.5
2021/12/09 07:17:41 Rebalance progress: 12.5
2021/12/09 07:17:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:17:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:17:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:17:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 07:17:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 07:18:01 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 07:18:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 07:18:06 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/09 07:18:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:18:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/09 07:18:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:18:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:18:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:18:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:18:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:18:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12182
2021/12/09 07:18:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:18:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:18:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 07:18:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 07:18:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 07:18:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 07:18:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/09 07:18:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:18:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/09 07:18:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:18:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:18:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:18:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:19:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:19:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.28s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/09 07:19:10 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 07:19:10 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/09 07:19:10 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:19:10 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/09 07:19:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:19:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:19:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:19:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:19:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:19:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:19:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:19:28 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:19:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:19:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:19:30 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:19:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 07:19:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:19:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 07:19:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 07:19:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 07:19:51 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/09 07:19:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 07:20:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 07:20:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 07:20:06 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:20:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:20:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:20:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:20:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:20:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 07:20:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:20:12 Rebalance progress: 12.5
2021/12/09 07:20:17 Rebalance progress: 12.5
2021/12/09 07:20:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:20:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:20:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:20:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 07:20:27 Rebalance progress: 12.5
2021/12/09 07:20:32 Rebalance progress: 12.5
2021/12/09 07:20:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:20:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:20:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:20:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:20:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 07:20:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 07:20:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 07:20:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 07:20:57 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/09 07:20:57 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:20:57 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/09 07:21:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:21:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 757
2021/12/09 07:21:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 757
2021/12/09 07:21:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:21:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:21:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:21:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6142
2021/12/09 07:21:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:21:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:21:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 07:21:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 07:21:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 07:21:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 07:21:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/09 07:21:47 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:21:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/09 07:21:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:21:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:21:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:21:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (180.73s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/09 07:22:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 07:22:11 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/09 07:22:11 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:22:11 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/09 07:22:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:22:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:22:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 07:22:29 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/09 07:22:29 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:22:29 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/09 07:22:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:22:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:43 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:22:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:22:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:22:44 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:22:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 07:22:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:22:55 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 07:23:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 07:23:05 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 07:23:05 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/09 07:23:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 07:23:15 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 07:23:20 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 07:23:20 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:23:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:23:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:23:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:23:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:23:25 Rebalance progress: 12.5
2021/12/09 07:23:30 Rebalance progress: 12.5
2021/12/09 07:23:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:23:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:23:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:23:41 Rebalance progress: 12.5
2021/12/09 07:23:46 Rebalance progress: 12.5
2021/12/09 07:23:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:23:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:23:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:23:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:23:56 Rebalance progress: 12.5
2021/12/09 07:24:01 Rebalance progress: 12.5
2021/12/09 07:24:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:24:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:24:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 07:24:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 07:24:21 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 07:24:26 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 07:24:26 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/09 07:24:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:24:26 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/09 07:24:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:24:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:24:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:24:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:24:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:24:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7594
2021/12/09 07:24:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:24:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:24:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 07:24:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 07:25:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 07:25:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 07:25:06 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/09 07:25:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:25:07 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/09 07:25:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:25:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:25:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:25:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:25:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:25:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (199.38s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 07:25:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:25:30 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 07:25:30 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:25:30 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 07:25:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:25:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:25:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:25:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:25:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:25:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:25:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:25:49 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:25:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 07:25:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:26:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:26:05 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:26:10 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:26:10 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:26:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:26:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:26:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:26:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:26:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:26:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:26:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:27:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:27:06 Rebalance progress: 12.5
2021/12/09 07:27:11 Rebalance progress: 12.5
2021/12/09 07:27:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:27:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:27:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:27:21 Rebalance progress: 12.5
2021/12/09 07:27:26 Rebalance progress: 12.5
2021/12/09 07:27:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:27:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:27:36 Rebalance progress: 12.5
2021/12/09 07:27:41 Rebalance progress: 12.5
2021/12/09 07:27:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:27:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:27:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:27:52 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:27:57 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:28:02 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:28:07 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:28:07 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 07:28:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:28:07 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 07:28:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:28:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:28:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11172
2021/12/09 07:28:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:28:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:28:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:28:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3183
2021/12/09 07:28:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3183
2021/12/09 07:28:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.16s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 07:28:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:28:45 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 07:28:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:28:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 07:28:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:28:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:28:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:28:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:28:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:28:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:29:04 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:29:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 07:29:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:29:15 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:29:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:29:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 07:29:25 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:29:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:29:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:29:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:29:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:29:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:29:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:30:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:30:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:30:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:30:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:30:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:30:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:31:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:31:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:31:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:31:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:31:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:31:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:32:04 Rebalance progress: 12.5
2021/12/09 07:32:09 Rebalance progress: 12.5
2021/12/09 07:32:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:32:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:32:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:32:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:32:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:32:20 Rebalance progress: 12.5
2021/12/09 07:32:25 Rebalance progress: 12.5
2021/12/09 07:32:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:32:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:32:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:32:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:32:35 Rebalance progress: 12.5
2021/12/09 07:32:40 Rebalance progress: 12.5
2021/12/09 07:32:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:32:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:32:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:32:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:32:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:32:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:33:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:33:05 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 07:33:05 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 07:33:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:33:05 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 07:33:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:33:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:33:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21486
2021/12/09 07:33:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:33:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:33:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:33:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21486
2021/12/09 07:33:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (293.32s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/09 07:33:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 07:33:39 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/09 07:33:39 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:33:39 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/09 07:33:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:33:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:33:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:33:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:33:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:33:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:33:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:33:57 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:33:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 07:34:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:34:08 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 07:34:13 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 07:34:18 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 07:34:18 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:34:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:34:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:34:23 Rebalance progress: 12.5
2021/12/09 07:34:28 Rebalance progress: 12.5
2021/12/09 07:34:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:34:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:34:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:34:48 Rebalance progress: 12.5
2021/12/09 07:34:53 Rebalance progress: 12.5
2021/12/09 07:34:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:34:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:34:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:34:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:35:04 Rebalance progress: 12.5
2021/12/09 07:35:09 Rebalance progress: 12.5
2021/12/09 07:35:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:35:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:35:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:35:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:35:19 Rebalance progress: 12.5
2021/12/09 07:35:24 Rebalance progress: 12.5
2021/12/09 07:35:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:35:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:35:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:35:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:35:34 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 07:35:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 07:35:44 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 07:35:49 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 07:35:49 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/09 07:35:49 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:35:49 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/09 07:35:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:35:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:36:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8289
2021/12/09 07:36:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8289
2021/12/09 07:36:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:36:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:36:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:36:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8574
2021/12/09 07:36:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.11s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/09 07:36:28 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 07:36:28 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/09 07:36:28 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:36:28 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/09 07:36:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:36:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:36:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:36:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:36:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:36:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:36:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:36:46 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:36:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 07:36:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:36:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 07:37:02 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 07:37:07 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 07:37:07 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:37:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/09 07:37:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:37:13 Rebalance progress: 12.5
2021/12/09 07:37:18 Rebalance progress: 12.5
2021/12/09 07:37:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:37:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/09 07:37:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:37:28 Rebalance progress: 12.5
2021/12/09 07:37:33 Rebalance progress: 12.5
2021/12/09 07:37:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:37:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:37:43 Rebalance progress: 12.5
2021/12/09 07:37:48 Rebalance progress: 12.5
2021/12/09 07:37:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:37:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:37:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:37:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 07:38:03 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 07:38:08 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 07:38:13 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 07:38:13 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/09 07:38:13 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:38:13 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/09 07:38:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:38:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:38:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6064
2021/12/09 07:38:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:38:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:38:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:38:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6064
2021/12/09 07:38:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.13s)
=== RUN   TestEventingRebInWithOps
2021/12/09 07:38:47 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 07:38:47 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/09 07:38:47 eventingSetup.go:46: No eventing-con process running
2021/12/09 07:38:47 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/09 07:38:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:38:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:38:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:38:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:39:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:39:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:39:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:39:05 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:39:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 07:39:11 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 07:39:16 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 07:39:21 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 07:39:26 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 07:39:26 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/09 07:39:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:39:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 07:39:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 07:39:32 Rebalance progress: 12.5
2021/12/09 07:39:37 Rebalance progress: 12.5
2021/12/09 07:39:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:39:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:39:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:39:47 Rebalance progress: 12.5
2021/12/09 07:39:52 Rebalance progress: 12.5
2021/12/09 07:39:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:39:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:39:57 Waiting for src bucket item count to get to: 200000 curr count: 35028
2021/12/09 07:40:02 Waiting for src bucket item count to get to: 200000 curr count: 51333
2021/12/09 07:40:07 Waiting for src bucket item count to get to: 200000 curr count: 51333
2021/12/09 07:40:12 Waiting for src bucket item count to get to: 200000 curr count: 67681
2021/12/09 07:40:17 Waiting for src bucket item count to get to: 200000 curr count: 67681
2021/12/09 07:40:22 Waiting for src bucket item count to get to: 200000 curr count: 82582
2021/12/09 07:40:27 Waiting for src bucket item count to get to: 200000 curr count: 82582
2021/12/09 07:40:32 Waiting for src bucket item count to get to: 200000 curr count: 98698
2021/12/09 07:40:37 Waiting for src bucket item count to get to: 200000 curr count: 98698
2021/12/09 07:40:42 Waiting for src bucket item count to get to: 200000 curr count: 114928
2021/12/09 07:40:47 Waiting for src bucket item count to get to: 200000 curr count: 114928
2021/12/09 07:40:52 Waiting for src bucket item count to get to: 200000 curr count: 114928
2021/12/09 07:40:57 Waiting for src bucket item count to get to: 200000 curr count: 130974
2021/12/09 07:41:02 Waiting for src bucket item count to get to: 200000 curr count: 130974
2021/12/09 07:41:07 Waiting for src bucket item count to get to: 200000 curr count: 147287
2021/12/09 07:41:12 Waiting for src bucket item count to get to: 200000 curr count: 147287
2021/12/09 07:41:17 Waiting for src bucket item count to get to: 200000 curr count: 163335
2021/12/09 07:41:22 Waiting for src bucket item count to get to: 200000 curr count: 163335
2021/12/09 07:41:27 Waiting for src bucket item count to get to: 200000 curr count: 178135
2021/12/09 07:41:32 Waiting for src bucket item count to get to: 200000 curr count: 178135
2021/12/09 07:41:37 Waiting for src bucket item count to get to: 200000 curr count: 194375
2021/12/09 07:41:42 Waiting for src bucket item count to get to: 200000 curr count: 194375
2021/12/09 07:41:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 07:41:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 07:41:47 Starting bucket ops to source bucket
2021/12/09 07:45:32 Waiting for dst bucket item count to get to: 0 curr count: 15091
2021/12/09 07:45:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:45:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 07:45:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:45:37 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 07:45:42 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 07:45:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 07:45:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 07:45:53 App: TestEventingRebInWithOps got un-deployed
2021/12/09 07:45:53 No eventing-con process running
2021/12/09 07:45:53 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/09 07:45:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:45:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:46:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:46:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.30s)
=== RUN   TestEventingRebOutWithOps
2021/12/09 07:46:11 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:46:11 Removing node(s): [] from the cluster
2021/12/09 07:46:17 Rebalance progress: 100
2021/12/09 07:46:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 07:46:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 07:46:22 No eventing-con process running
2021/12/09 07:46:22 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/09 07:46:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:46:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:46:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:46:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:46:40 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:46:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 07:46:46 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 07:46:51 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 07:46:56 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 07:47:01 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 07:47:01 App: TestEventingRebOutWithOps got deployed
2021/12/09 07:47:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:47:01 Starting bucket ops to source bucket
2021/12/09 07:47:06 Rebalance progress: 12.5
2021/12/09 07:47:11 Rebalance progress: 12.5
2021/12/09 07:47:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:47:16 Waiting for src bucket item count to get to: 200000 curr count: 12543
2021/12/09 07:47:21 Waiting for src bucket item count to get to: 200000 curr count: 28599
2021/12/09 07:47:26 Waiting for src bucket item count to get to: 200000 curr count: 28599
2021/12/09 07:47:31 Waiting for src bucket item count to get to: 200000 curr count: 45087
2021/12/09 07:47:36 Waiting for src bucket item count to get to: 200000 curr count: 45087
2021/12/09 07:47:41 Waiting for src bucket item count to get to: 200000 curr count: 61255
2021/12/09 07:47:46 Waiting for src bucket item count to get to: 200000 curr count: 61255
2021/12/09 07:47:51 Waiting for src bucket item count to get to: 200000 curr count: 61255
2021/12/09 07:47:56 Waiting for src bucket item count to get to: 200000 curr count: 76594
2021/12/09 07:48:01 Waiting for src bucket item count to get to: 200000 curr count: 76594
2021/12/09 07:48:06 Waiting for src bucket item count to get to: 200000 curr count: 92597
2021/12/09 07:48:11 Waiting for src bucket item count to get to: 200000 curr count: 92597
2021/12/09 07:48:16 Waiting for src bucket item count to get to: 200000 curr count: 108915
2021/12/09 07:48:21 Waiting for src bucket item count to get to: 200000 curr count: 108915
2021/12/09 07:48:26 Waiting for src bucket item count to get to: 200000 curr count: 125188
2021/12/09 07:48:31 Waiting for src bucket item count to get to: 200000 curr count: 125188
2021/12/09 07:48:36 Waiting for src bucket item count to get to: 200000 curr count: 141298
2021/12/09 07:48:41 Waiting for src bucket item count to get to: 200000 curr count: 141298
2021/12/09 07:48:46 Waiting for src bucket item count to get to: 200000 curr count: 157575
2021/12/09 07:48:51 Waiting for src bucket item count to get to: 200000 curr count: 157575
2021/12/09 07:48:56 Waiting for src bucket item count to get to: 200000 curr count: 172681
2021/12/09 07:49:01 Waiting for src bucket item count to get to: 200000 curr count: 172681
2021/12/09 07:49:06 Waiting for src bucket item count to get to: 200000 curr count: 188439
2021/12/09 07:49:11 Waiting for src bucket item count to get to: 200000 curr count: 188439
2021/12/09 07:49:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 07:49:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 07:49:16 Starting bucket ops to source bucket
2021/12/09 07:53:01 Waiting for dst bucket item count to get to: 0 curr count: 13049
2021/12/09 07:53:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:53:01 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 07:53:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 07:53:06 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 07:53:12 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 07:53:17 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 07:53:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 07:53:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 07:53:22 No eventing-con process running
2021/12/09 07:53:22 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/09 07:53:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:53:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:53:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (448.99s)
=== RUN   TestEventingSwapRebWithOps
2021/12/09 07:53:45 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 07:53:45 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 07:53:45 No eventing-con process running
2021/12/09 07:53:45 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/09 07:53:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:53:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:53:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 07:53:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:53:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:54:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 07:54:04 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 07:54:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 07:54:10 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 07:54:15 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 07:54:20 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 07:54:25 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 07:54:25 App: TestEventingSwapRebWithOps got deployed
2021/12/09 07:54:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 07:54:25 Starting bucket ops to source bucket
2021/12/09 07:54:25 Removing node(s): [] from the cluster
2021/12/09 07:54:30 Rebalance progress: 12.5
2021/12/09 07:54:35 Rebalance progress: 12.5
2021/12/09 07:54:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:54:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:54:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:54:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 07:54:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 07:54:46 Rebalance progress: 12.5
2021/12/09 07:54:51 Rebalance progress: 12.5
2021/12/09 07:54:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:54:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 07:55:01 Rebalance progress: 12.5
2021/12/09 07:55:06 Rebalance progress: 12.5
2021/12/09 07:55:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:55:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 07:55:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 07:55:11 Waiting for src bucket item count to get to: 200000 curr count: 69905
2021/12/09 07:55:16 Waiting for src bucket item count to get to: 200000 curr count: 69905
2021/12/09 07:55:21 Waiting for src bucket item count to get to: 200000 curr count: 85014
2021/12/09 07:55:26 Waiting for src bucket item count to get to: 200000 curr count: 85014
2021/12/09 07:55:31 Waiting for src bucket item count to get to: 200000 curr count: 101543
2021/12/09 07:55:36 Waiting for src bucket item count to get to: 200000 curr count: 101543
2021/12/09 07:55:41 Waiting for src bucket item count to get to: 200000 curr count: 118027
2021/12/09 07:55:46 Waiting for src bucket item count to get to: 200000 curr count: 118027
2021/12/09 07:55:51 Waiting for src bucket item count to get to: 200000 curr count: 134298
2021/12/09 07:55:56 Waiting for src bucket item count to get to: 200000 curr count: 134298
2021/12/09 07:56:01 Waiting for src bucket item count to get to: 200000 curr count: 150379
2021/12/09 07:56:06 Waiting for src bucket item count to get to: 200000 curr count: 150379
2021/12/09 07:56:11 Waiting for src bucket item count to get to: 200000 curr count: 166430
2021/12/09 07:56:16 Waiting for src bucket item count to get to: 200000 curr count: 166430
2021/12/09 07:56:21 Waiting for src bucket item count to get to: 200000 curr count: 181223
2021/12/09 07:56:26 Waiting for src bucket item count to get to: 200000 curr count: 181223
2021/12/09 07:56:31 Waiting for src bucket item count to get to: 200000 curr count: 197510
2021/12/09 07:56:36 Waiting for src bucket item count to get to: 200000 curr count: 197510
2021/12/09 07:56:41 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 07:56:41 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 07:56:41 Starting bucket ops to source bucket
2021/12/09 08:00:28 Waiting for dst bucket item count to get to: 0 curr count: 5943
2021/12/09 08:00:33 Waiting for dst bucket item count to get to: 0 curr count: 5943
2021/12/09 08:00:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:00:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:00:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:00:38 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 08:00:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 08:00:48 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 08:00:53 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 08:00:53 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 08:00:53 No eventing-con process running
2021/12/09 08:00:53 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/09 08:00:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:00:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:00:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:01:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:01:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (451.18s)
=== RUN   TestKVRebInWithOps
2021/12/09 08:01:16 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 08:01:16 App: TestKVRebInWithOps got un-deployed
2021/12/09 08:01:16 No eventing-con process running
2021/12/09 08:01:16 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/09 08:01:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:01:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:01:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:01:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:01:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:01:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:01:35 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:01:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 08:01:41 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 08:01:46 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 08:01:51 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 08:01:56 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 08:01:56 App: TestKVRebInWithOps got deployed
2021/12/09 08:01:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:01:56 Starting bucket ops to source bucket
2021/12/09 08:01:57 Removing node(s): [] from the cluster
2021/12/09 08:02:02 Rebalance progress: 12.5
2021/12/09 08:02:07 Rebalance progress: 12.5
2021/12/09 08:02:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:02:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:02:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:02:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:02:17 Rebalance progress: 12.5
2021/12/09 08:02:22 Rebalance progress: 12.5
2021/12/09 08:02:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:02:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:02:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:02:27 Waiting for src bucket item count to get to: 200000 curr count: 36085
2021/12/09 08:02:32 Waiting for src bucket item count to get to: 200000 curr count: 52615
2021/12/09 08:02:37 Waiting for src bucket item count to get to: 200000 curr count: 52615
2021/12/09 08:02:42 Waiting for src bucket item count to get to: 200000 curr count: 69205
2021/12/09 08:02:47 Waiting for src bucket item count to get to: 200000 curr count: 69205
2021/12/09 08:02:52 Waiting for src bucket item count to get to: 200000 curr count: 84455
2021/12/09 08:02:57 Waiting for src bucket item count to get to: 200000 curr count: 84455
2021/12/09 08:03:02 Waiting for src bucket item count to get to: 200000 curr count: 100963
2021/12/09 08:03:07 Waiting for src bucket item count to get to: 200000 curr count: 100963
2021/12/09 08:03:12 Waiting for src bucket item count to get to: 200000 curr count: 117411
2021/12/09 08:03:17 Waiting for src bucket item count to get to: 200000 curr count: 117411
2021/12/09 08:03:22 Waiting for src bucket item count to get to: 200000 curr count: 133820
2021/12/09 08:03:27 Waiting for src bucket item count to get to: 200000 curr count: 133820
2021/12/09 08:03:32 Waiting for src bucket item count to get to: 200000 curr count: 150355
2021/12/09 08:03:37 Waiting for src bucket item count to get to: 200000 curr count: 150355
2021/12/09 08:03:42 Waiting for src bucket item count to get to: 200000 curr count: 166891
2021/12/09 08:03:47 Waiting for src bucket item count to get to: 200000 curr count: 166891
2021/12/09 08:03:52 Waiting for src bucket item count to get to: 200000 curr count: 182084
2021/12/09 08:03:57 Waiting for src bucket item count to get to: 200000 curr count: 182084
2021/12/09 08:04:02 Waiting for src bucket item count to get to: 200000 curr count: 182084
2021/12/09 08:04:07 Waiting for src bucket item count to get to: 200000 curr count: 198661
2021/12/09 08:04:12 Waiting for src bucket item count to get to: 200000 curr count: 198661
2021/12/09 08:04:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 08:04:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 08:04:17 Starting bucket ops to source bucket
2021/12/09 08:08:02 Waiting for dst bucket item count to get to: 0 curr count: 14963
2021/12/09 08:08:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:08:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:08:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:08:07 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 08:08:12 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 08:08:17 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 08:08:22 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 08:08:22 App: TestKVRebInWithOps got un-deployed
2021/12/09 08:08:22 No eventing-con process running
2021/12/09 08:08:23 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/09 08:08:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:08:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:08:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:08:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:08:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.78s)
=== RUN   TestKVRebOutWithOps
2021/12/09 08:08:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:08:42 Removing node(s): [] from the cluster
2021/12/09 08:08:47 Rebalance progress: 100
2021/12/09 08:08:52 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 08:08:52 App: TestKVRebOutWithOps got un-deployed
2021/12/09 08:08:52 No eventing-con process running
2021/12/09 08:08:52 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/09 08:08:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:08:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:08:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:09:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:09:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:09:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:09:11 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:09:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 08:09:17 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 08:09:22 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 08:09:27 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 08:09:32 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 08:09:32 App: TestKVRebOutWithOps got deployed
2021/12/09 08:09:32 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:09:32 Starting bucket ops to source bucket
2021/12/09 08:09:37 Rebalance progress: 12.5
2021/12/09 08:09:42 Rebalance progress: 12.5
2021/12/09 08:09:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:09:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:09:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:09:47 Waiting for src bucket item count to get to: 200000 curr count: 10984
2021/12/09 08:09:52 Waiting for src bucket item count to get to: 200000 curr count: 27169
2021/12/09 08:09:57 Waiting for src bucket item count to get to: 200000 curr count: 27169
2021/12/09 08:10:02 Waiting for src bucket item count to get to: 200000 curr count: 43345
2021/12/09 08:10:07 Waiting for src bucket item count to get to: 200000 curr count: 43345
2021/12/09 08:10:12 Waiting for src bucket item count to get to: 200000 curr count: 59617
2021/12/09 08:10:17 Waiting for src bucket item count to get to: 200000 curr count: 59617
2021/12/09 08:10:22 Waiting for src bucket item count to get to: 200000 curr count: 59617
2021/12/09 08:10:27 Waiting for src bucket item count to get to: 200000 curr count: 74752
2021/12/09 08:10:32 Waiting for src bucket item count to get to: 200000 curr count: 74752
2021/12/09 08:10:37 Waiting for src bucket item count to get to: 200000 curr count: 90547
2021/12/09 08:10:42 Waiting for src bucket item count to get to: 200000 curr count: 90547
2021/12/09 08:10:47 Waiting for src bucket item count to get to: 200000 curr count: 106931
2021/12/09 08:10:52 Waiting for src bucket item count to get to: 200000 curr count: 106931
2021/12/09 08:10:57 Waiting for src bucket item count to get to: 200000 curr count: 123160
2021/12/09 08:11:02 Waiting for src bucket item count to get to: 200000 curr count: 123160
2021/12/09 08:11:07 Waiting for src bucket item count to get to: 200000 curr count: 139357
2021/12/09 08:11:12 Waiting for src bucket item count to get to: 200000 curr count: 139357
2021/12/09 08:11:17 Waiting for src bucket item count to get to: 200000 curr count: 155657
2021/12/09 08:11:22 Waiting for src bucket item count to get to: 200000 curr count: 155657
2021/12/09 08:11:27 Waiting for src bucket item count to get to: 200000 curr count: 171065
2021/12/09 08:11:32 Waiting for src bucket item count to get to: 200000 curr count: 171065
2021/12/09 08:11:37 Waiting for src bucket item count to get to: 200000 curr count: 187297
2021/12/09 08:11:42 Waiting for src bucket item count to get to: 200000 curr count: 187297
2021/12/09 08:11:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 08:11:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 08:11:47 Starting bucket ops to source bucket
2021/12/09 08:15:32 Waiting for dst bucket item count to get to: 0 curr count: 14377
2021/12/09 08:15:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:15:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:15:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:15:37 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 08:15:42 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 08:15:47 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 08:15:52 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 08:15:52 App: TestKVRebOutWithOps got un-deployed
2021/12/09 08:15:52 No eventing-con process running
2021/12/09 08:15:52 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/09 08:15:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:15:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:15:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:16:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (449.80s)
=== RUN   TestKVSwapRebWithOps
2021/12/09 08:16:16 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 08:16:16 App: TestKVSwapRebWithOps got un-deployed
2021/12/09 08:16:16 No eventing-con process running
2021/12/09 08:16:16 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/09 08:16:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:16:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:16:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:16:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:16:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:16:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:16:34 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:16:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 08:16:40 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 08:16:45 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 08:16:50 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 08:16:55 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/09 08:16:55 App: TestKVSwapRebWithOps got deployed
2021/12/09 08:16:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:16:55 Starting bucket ops to source bucket
2021/12/09 08:16:56 Removing node(s): [] from the cluster
2021/12/09 08:17:01 Rebalance progress: 12.5
2021/12/09 08:17:06 Rebalance progress: 12.5
2021/12/09 08:17:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:17:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:17:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:17:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 08:17:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:17:16 Rebalance progress: 12.5
2021/12/09 08:17:21 Rebalance progress: 12.5
2021/12/09 08:17:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:17:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:17:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:17:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 08:17:31 Rebalance progress: 12.5
2021/12/09 08:17:36 Rebalance progress: 12.5
2021/12/09 08:17:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:17:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:17:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:17:41 Waiting for src bucket item count to get to: 200000 curr count: 68843
2021/12/09 08:17:46 Waiting for src bucket item count to get to: 200000 curr count: 68843
2021/12/09 08:17:51 Waiting for src bucket item count to get to: 200000 curr count: 83655
2021/12/09 08:17:56 Waiting for src bucket item count to get to: 200000 curr count: 83655
2021/12/09 08:18:01 Waiting for src bucket item count to get to: 200000 curr count: 100241
2021/12/09 08:18:06 Waiting for src bucket item count to get to: 200000 curr count: 100241
2021/12/09 08:18:11 Waiting for src bucket item count to get to: 200000 curr count: 116751
2021/12/09 08:18:17 Waiting for src bucket item count to get to: 200000 curr count: 116751
2021/12/09 08:18:22 Waiting for src bucket item count to get to: 200000 curr count: 133329
2021/12/09 08:18:27 Waiting for src bucket item count to get to: 200000 curr count: 133329
2021/12/09 08:18:32 Waiting for src bucket item count to get to: 200000 curr count: 149714
2021/12/09 08:18:37 Waiting for src bucket item count to get to: 200000 curr count: 149714
2021/12/09 08:18:42 Waiting for src bucket item count to get to: 200000 curr count: 166179
2021/12/09 08:18:47 Waiting for src bucket item count to get to: 200000 curr count: 166179
2021/12/09 08:18:52 Waiting for src bucket item count to get to: 200000 curr count: 166179
2021/12/09 08:18:57 Waiting for src bucket item count to get to: 200000 curr count: 181378
2021/12/09 08:19:02 Waiting for src bucket item count to get to: 200000 curr count: 181378
2021/12/09 08:19:07 Waiting for src bucket item count to get to: 200000 curr count: 197698
2021/12/09 08:19:12 Waiting for src bucket item count to get to: 200000 curr count: 197698
2021/12/09 08:19:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 08:19:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 08:19:17 Starting bucket ops to source bucket
2021/12/09 08:23:02 Waiting for dst bucket item count to get to: 0 curr count: 14493
2021/12/09 08:23:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:23:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:23:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:23:07 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 08:23:12 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 08:23:17 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 08:23:22 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/09 08:23:22 App: TestKVSwapRebWithOps got un-deployed
2021/12/09 08:23:22 No eventing-con process running
2021/12/09 08:23:22 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/09 08:23:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:23:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:23:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:23:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:23:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:23:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (449.93s)
=== RUN   TestEventingRebInWithTimers
2021/12/09 08:23:46 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 08:23:46 App: TestEventingRebInWithTimers got un-deployed
2021/12/09 08:23:46 No eventing-con process running
2021/12/09 08:23:46 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/09 08:23:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:23:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:23:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:23:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:23:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:24:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:24:04 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:24:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 08:24:10 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 08:24:15 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 08:24:20 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 08:24:25 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/09 08:24:25 App: TestEventingRebInWithTimers got deployed
2021/12/09 08:24:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 08:24:25 Starting bucket ops to source bucket
2021/12/09 08:24:26 Removing node(s): [] from the cluster
2021/12/09 08:24:31 Rebalance progress: 12.5
2021/12/09 08:24:36 Rebalance progress: 12.5
2021/12/09 08:24:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:24:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:24:46 Rebalance progress: 12.5
2021/12/09 08:24:51 Rebalance progress: 12.5
2021/12/09 08:24:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:24:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:24:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:24:57 Waiting for src bucket item count to get to: 50000 curr count: 27532
2021/12/09 08:25:02 Waiting for src bucket item count to get to: 50000 curr count: 40197
2021/12/09 08:25:07 Waiting for src bucket item count to get to: 50000 curr count: 40197
2021/12/09 08:25:17 Waiting for dst bucket item count to get to: 50000 curr count: 40692
2021/12/09 08:25:22 Waiting for dst bucket item count to get to: 50000 curr count: 40692
2021/12/09 08:25:27 Waiting for dst bucket item count to get to: 50000 curr count: 48894
2021/12/09 08:25:32 Waiting for dst bucket item count to get to: 50000 curr count: 48894
2021/12/09 08:25:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:25:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:25:32 Starting bucket ops to source bucket
2021/12/09 08:26:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:26:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:26:51 Waiting for dst bucket item count to get to: 0 curr count: 44364
2021/12/09 08:26:56 Waiting for dst bucket item count to get to: 0 curr count: 44364
2021/12/09 08:27:01 Waiting for dst bucket item count to get to: 0 curr count: 35261
2021/12/09 08:27:06 Waiting for dst bucket item count to get to: 0 curr count: 35261
2021/12/09 08:27:11 Waiting for dst bucket item count to get to: 0 curr count: 21118
2021/12/09 08:27:16 Waiting for dst bucket item count to get to: 0 curr count: 21118
2021/12/09 08:27:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:27:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:27:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:27:22 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 08:27:27 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 08:27:32 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 08:27:37 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/09 08:27:37 App: TestEventingRebInWithTimers got un-deployed
2021/12/09 08:27:37 No eventing-con process running
2021/12/09 08:27:37 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/09 08:27:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:27:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:27:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:27:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:27:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:27:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (254.40s)
=== RUN   TestEventingRebOutWithTimers
2021/12/09 08:27:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 08:27:55 Removing node(s): [] from the cluster
2021/12/09 08:28:01 Rebalance progress: 100
2021/12/09 08:28:06 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 08:28:06 App: TestEventingRebOutWithTimers got un-deployed
2021/12/09 08:28:06 No eventing-con process running
2021/12/09 08:28:06 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/09 08:28:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:28:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:28:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:28:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:28:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:28:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:28:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:28:24 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:28:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 08:28:30 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 08:28:35 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 08:28:40 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 08:28:45 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/09 08:28:45 App: TestEventingRebOutWithTimers got deployed
2021/12/09 08:28:45 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:28:45 Starting bucket ops to source bucket
2021/12/09 08:28:50 Rebalance progress: 12.5
2021/12/09 08:28:55 Rebalance progress: 12.5
2021/12/09 08:29:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:29:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:29:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:29:00 Waiting for src bucket item count to get to: 50000 curr count: 19340
2021/12/09 08:29:05 Waiting for src bucket item count to get to: 50000 curr count: 19340
2021/12/09 08:29:10 Waiting for src bucket item count to get to: 50000 curr count: 29239
2021/12/09 08:29:15 Waiting for src bucket item count to get to: 50000 curr count: 29239
2021/12/09 08:29:20 Waiting for src bucket item count to get to: 50000 curr count: 41886
2021/12/09 08:29:25 Waiting for src bucket item count to get to: 50000 curr count: 41886
2021/12/09 08:29:35 Waiting for dst bucket item count to get to: 50000 curr count: 41997
2021/12/09 08:29:40 Waiting for dst bucket item count to get to: 50000 curr count: 41997
2021/12/09 08:29:45 Waiting for dst bucket item count to get to: 50000 curr count: 41997
2021/12/09 08:29:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:29:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:29:45 Starting bucket ops to source bucket
2021/12/09 08:30:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:31:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:31:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:31:10 Waiting for dst bucket item count to get to: 0 curr count: 39692
2021/12/09 08:31:15 Waiting for dst bucket item count to get to: 0 curr count: 39692
2021/12/09 08:31:20 Waiting for dst bucket item count to get to: 0 curr count: 26242
2021/12/09 08:31:25 Waiting for dst bucket item count to get to: 0 curr count: 26242
2021/12/09 08:31:30 Waiting for dst bucket item count to get to: 0 curr count: 10690
2021/12/09 08:31:35 Waiting for dst bucket item count to get to: 0 curr count: 10690
2021/12/09 08:31:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:31:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:31:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:31:40 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 08:31:45 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 08:31:50 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 08:31:55 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/09 08:31:55 App: TestEventingRebOutWithTimers got un-deployed
2021/12/09 08:31:55 No eventing-con process running
2021/12/09 08:31:55 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/09 08:31:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:32:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:32:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:32:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:32:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (258.29s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/09 08:32:18 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 08:32:18 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/09 08:32:18 No eventing-con process running
2021/12/09 08:32:18 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/09 08:32:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:32:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:32:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:32:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:32:37 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:32:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 08:32:43 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 08:32:48 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 08:32:53 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 08:32:58 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/09 08:32:58 App: TestEventingSwapRebWithTimers got deployed
2021/12/09 08:32:58 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 08:32:58 Starting bucket ops to source bucket
2021/12/09 08:32:58 Removing node(s): [] from the cluster
2021/12/09 08:33:03 Rebalance progress: 12.5
2021/12/09 08:33:08 Rebalance progress: 12.5
2021/12/09 08:33:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:33:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:33:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 08:33:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:33:19 Rebalance progress: 12.5
2021/12/09 08:33:24 Rebalance progress: 12.5
2021/12/09 08:33:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:33:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:33:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:33:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 08:33:34 Rebalance progress: 12.5
2021/12/09 08:33:39 Rebalance progress: 12.5
2021/12/09 08:33:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:33:44 Waiting for src bucket item count to get to: 50000 curr count: 48614
2021/12/09 08:33:49 Waiting for src bucket item count to get to: 50000 curr count: 48614
2021/12/09 08:33:59 Waiting for dst bucket item count to get to: 50000 curr count: 47445
2021/12/09 08:34:04 Waiting for dst bucket item count to get to: 50000 curr count: 47445
2021/12/09 08:34:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:34:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:34:04 Starting bucket ops to source bucket
2021/12/09 08:35:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:35:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:35:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:35:28 Waiting for dst bucket item count to get to: 0 curr count: 41464
2021/12/09 08:35:33 Waiting for dst bucket item count to get to: 0 curr count: 41464
2021/12/09 08:35:38 Waiting for dst bucket item count to get to: 0 curr count: 26028
2021/12/09 08:35:43 Waiting for dst bucket item count to get to: 0 curr count: 26028
2021/12/09 08:35:48 Waiting for dst bucket item count to get to: 0 curr count: 9705
2021/12/09 08:35:53 Waiting for dst bucket item count to get to: 0 curr count: 9705
2021/12/09 08:35:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:35:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:35:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:35:59 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 08:36:04 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 08:36:09 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 08:36:14 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/09 08:36:14 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/09 08:36:14 No eventing-con process running
2021/12/09 08:36:14 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/09 08:36:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:36:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:36:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:36:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:36:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (258.78s)
=== RUN   TestKVRebInWithTimers
2021/12/09 08:36:37 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 08:36:37 App: TestKVRebInWithTimers got un-deployed
2021/12/09 08:36:37 No eventing-con process running
2021/12/09 08:36:37 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/09 08:36:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:36:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:36:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:36:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:36:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:36:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:36:56 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:36:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 08:37:02 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 08:37:07 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 08:37:12 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 08:37:17 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/09 08:37:17 App: TestKVRebInWithTimers got deployed
2021/12/09 08:37:17 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:37:17 Starting bucket ops to source bucket
2021/12/09 08:37:17 Removing node(s): [] from the cluster
2021/12/09 08:37:22 Rebalance progress: 12.5
2021/12/09 08:37:27 Rebalance progress: 12.5
2021/12/09 08:37:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:37:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:37:37 Rebalance progress: 12.5
2021/12/09 08:37:42 Rebalance progress: 12.5
2021/12/09 08:37:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:37:47 Waiting for src bucket item count to get to: 50000 curr count: 27000
2021/12/09 08:37:52 Waiting for src bucket item count to get to: 50000 curr count: 38147
2021/12/09 08:37:57 Waiting for src bucket item count to get to: 50000 curr count: 38147
2021/12/09 08:38:02 Waiting for src bucket item count to get to: 50000 curr count: 49661
2021/12/09 08:38:08 Waiting for src bucket item count to get to: 50000 curr count: 49661
2021/12/09 08:38:18 Waiting for dst bucket item count to get to: 50000 curr count: 47994
2021/12/09 08:38:23 Waiting for dst bucket item count to get to: 50000 curr count: 47994
2021/12/09 08:38:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:38:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:38:23 Starting bucket ops to source bucket
2021/12/09 08:39:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:39:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:39:42 Waiting for dst bucket item count to get to: 0 curr count: 49172
2021/12/09 08:39:47 Waiting for dst bucket item count to get to: 0 curr count: 49172
2021/12/09 08:39:52 Waiting for dst bucket item count to get to: 0 curr count: 39379
2021/12/09 08:39:57 Waiting for dst bucket item count to get to: 0 curr count: 39379
2021/12/09 08:40:02 Waiting for dst bucket item count to get to: 0 curr count: 24648
2021/12/09 08:40:07 Waiting for dst bucket item count to get to: 0 curr count: 24648
2021/12/09 08:40:12 Waiting for dst bucket item count to get to: 0 curr count: 3579
2021/12/09 08:40:17 Waiting for dst bucket item count to get to: 0 curr count: 3579
2021/12/09 08:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:40:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:40:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:40:22 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 08:40:27 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 08:40:32 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 08:40:37 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/09 08:40:37 App: TestKVRebInWithTimers got un-deployed
2021/12/09 08:40:37 No eventing-con process running
2021/12/09 08:40:37 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/09 08:40:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:40:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:40:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:40:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:40:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (263.19s)
=== RUN   TestKVRebOutWithTimers
2021/12/09 08:40:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:40:56 Removing node(s): [] from the cluster
2021/12/09 08:41:01 Rebalance progress: 100
2021/12/09 08:41:06 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 08:41:06 App: TestKVRebOutWithTimers got un-deployed
2021/12/09 08:41:06 No eventing-con process running
2021/12/09 08:41:06 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/09 08:41:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:41:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:41:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:41:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:41:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:41:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:41:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:41:24 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:41:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 08:41:30 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 08:41:35 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 08:41:40 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 08:41:45 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/09 08:41:45 App: TestKVRebOutWithTimers got deployed
2021/12/09 08:41:45 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:41:45 Starting bucket ops to source bucket
2021/12/09 08:41:50 Rebalance progress: 12.5
2021/12/09 08:41:55 Rebalance progress: 12.5
2021/12/09 08:42:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:42:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:42:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:42:00 Waiting for src bucket item count to get to: 50000 curr count: 19734
2021/12/09 08:42:05 Waiting for src bucket item count to get to: 50000 curr count: 19734
2021/12/09 08:42:10 Waiting for src bucket item count to get to: 50000 curr count: 31036
2021/12/09 08:42:15 Waiting for src bucket item count to get to: 50000 curr count: 31036
2021/12/09 08:42:20 Waiting for src bucket item count to get to: 50000 curr count: 42489
2021/12/09 08:42:25 Waiting for src bucket item count to get to: 50000 curr count: 42489
2021/12/09 08:42:30 Waiting for src bucket item count to get to: 50000 curr count: 42489
2021/12/09 08:42:40 Waiting for dst bucket item count to get to: 50000 curr count: 38458
2021/12/09 08:42:46 Waiting for dst bucket item count to get to: 50000 curr count: 38458
2021/12/09 08:42:51 Waiting for dst bucket item count to get to: 50000 curr count: 46642
2021/12/09 08:42:56 Waiting for dst bucket item count to get to: 50000 curr count: 46642
2021/12/09 08:42:56 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:42:56 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:42:56 Starting bucket ops to source bucket
2021/12/09 08:44:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:44:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:44:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:44:20 Waiting for dst bucket item count to get to: 0 curr count: 47572
2021/12/09 08:44:25 Waiting for dst bucket item count to get to: 0 curr count: 47572
2021/12/09 08:44:30 Waiting for dst bucket item count to get to: 0 curr count: 34595
2021/12/09 08:44:35 Waiting for dst bucket item count to get to: 0 curr count: 34595
2021/12/09 08:44:40 Waiting for dst bucket item count to get to: 0 curr count: 13255
2021/12/09 08:44:45 Waiting for dst bucket item count to get to: 0 curr count: 13255
2021/12/09 08:44:50 Waiting for dst bucket item count to get to: 0 curr count: 1032
2021/12/09 08:44:55 Waiting for dst bucket item count to get to: 0 curr count: 1032
2021/12/09 08:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:44:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:44:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:45:00 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 08:45:05 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 08:45:10 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 08:45:15 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/09 08:45:15 App: TestKVRebOutWithTimers got un-deployed
2021/12/09 08:45:15 No eventing-con process running
2021/12/09 08:45:15 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/09 08:45:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:45:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:45:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (278.50s)
=== RUN   TestKVSwapRebWithTimers
2021/12/09 08:45:39 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:45:39 App: TestKVSwapRebWithTimers got un-deployed
2021/12/09 08:45:39 No eventing-con process running
2021/12/09 08:45:39 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/09 08:45:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:45:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:45:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:45:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:45:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:45:57 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:45:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/09 08:46:04 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 08:46:09 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 08:46:14 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 08:46:19 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/09 08:46:19 App: TestKVSwapRebWithTimers got deployed
2021/12/09 08:46:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:46:19 Starting bucket ops to source bucket
2021/12/09 08:46:19 Removing node(s): [] from the cluster
2021/12/09 08:46:25 Rebalance progress: 12.5
2021/12/09 08:46:30 Rebalance progress: 12.5
2021/12/09 08:46:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:46:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:46:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:46:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 08:46:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:46:40 Rebalance progress: 12.5
2021/12/09 08:46:45 Rebalance progress: 12.5
2021/12/09 08:46:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:46:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:46:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:46:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 08:46:55 Rebalance progress: 12.5
2021/12/09 08:47:00 Rebalance progress: 12.5
2021/12/09 08:47:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:47:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:47:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:47:05 Waiting for src bucket item count to get to: 50000 curr count: 49137
2021/12/09 08:47:10 Waiting for src bucket item count to get to: 50000 curr count: 49137
2021/12/09 08:47:20 Waiting for dst bucket item count to get to: 50000 curr count: 44955
2021/12/09 08:47:25 Waiting for dst bucket item count to get to: 50000 curr count: 44955
2021/12/09 08:47:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:47:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:47:25 Starting bucket ops to source bucket
2021/12/09 08:48:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:48:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:48:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:48:50 Waiting for dst bucket item count to get to: 0 curr count: 43880
2021/12/09 08:48:55 Waiting for dst bucket item count to get to: 0 curr count: 43880
2021/12/09 08:49:00 Waiting for dst bucket item count to get to: 0 curr count: 31568
2021/12/09 08:49:05 Waiting for dst bucket item count to get to: 0 curr count: 31568
2021/12/09 08:49:10 Waiting for dst bucket item count to get to: 0 curr count: 17165
2021/12/09 08:49:15 Waiting for dst bucket item count to get to: 0 curr count: 17165
2021/12/09 08:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:49:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 08:49:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:49:20 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:49:25 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:49:30 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:49:35 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:49:40 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:49:45 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/09 08:49:45 App: TestKVSwapRebWithTimers got un-deployed
2021/12/09 08:49:45 No eventing-con process running
2021/12/09 08:49:45 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/09 08:49:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:49:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:49:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:49:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:49:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:49:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (269.67s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/09 08:50:09 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:50:09 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/09 08:50:09 No eventing-con process running
2021/12/09 08:50:09 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/09 08:50:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:50:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:50:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:50:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:50:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:50:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:50:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:50:27 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:50:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 08:50:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 08:50:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 08:50:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 08:50:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/09 08:50:48 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/09 08:50:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 08:50:48 Starting bucket ops to source bucket
2021/12/09 08:50:49 Removing node(s): [] from the cluster
2021/12/09 08:50:54 Rebalance progress: 12.5
2021/12/09 08:50:59 Rebalance progress: 12.5
2021/12/09 08:51:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:51:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:51:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:51:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:51:09 Rebalance progress: 12.5
2021/12/09 08:51:14 Rebalance progress: 12.5
2021/12/09 08:51:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:51:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:51:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:51:19 Waiting for src bucket item count to get to: 50000 curr count: 27540
2021/12/09 08:51:24 Waiting for src bucket item count to get to: 50000 curr count: 37715
2021/12/09 08:51:29 Waiting for src bucket item count to get to: 50000 curr count: 37715
2021/12/09 08:51:34 Waiting for src bucket item count to get to: 50000 curr count: 48915
2021/12/09 08:51:39 Waiting for src bucket item count to get to: 50000 curr count: 48915
2021/12/09 08:51:49 Waiting for dst bucket item count to get to: 50000 curr count: 40342
2021/12/09 08:51:54 Waiting for dst bucket item count to get to: 50000 curr count: 40342
2021/12/09 08:51:59 Waiting for dst bucket item count to get to: 50000 curr count: 47777
2021/12/09 08:52:04 Waiting for dst bucket item count to get to: 50000 curr count: 47777
2021/12/09 08:52:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:52:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:52:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:52:09 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:52:14 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:52:19 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:52:24 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:52:29 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:52:34 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/09 08:52:34 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/09 08:52:34 No eventing-con process running
2021/12/09 08:52:34 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/09 08:52:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:52:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:52:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:52:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:52:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:52:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:52:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:52:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.25s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/09 08:53:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 08:53:03 Removing node(s): [] from the cluster
2021/12/09 08:53:08 Rebalance progress: 100
2021/12/09 08:53:13 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:53:13 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/09 08:53:13 No eventing-con process running
2021/12/09 08:53:13 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/09 08:53:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:53:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:53:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:53:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:53:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:53:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:53:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:53:32 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:53:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 08:53:38 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 08:53:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 08:53:48 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 08:53:53 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/09 08:53:53 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/09 08:53:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:53:53 Starting bucket ops to source bucket
2021/12/09 08:53:58 Rebalance progress: 12.5
2021/12/09 08:54:03 Rebalance progress: 12.5
2021/12/09 08:54:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:54:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:54:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:54:08 Waiting for src bucket item count to get to: 50000 curr count: 8038
2021/12/09 08:54:13 Waiting for src bucket item count to get to: 50000 curr count: 21757
2021/12/09 08:54:18 Waiting for src bucket item count to get to: 50000 curr count: 21757
2021/12/09 08:54:23 Waiting for src bucket item count to get to: 50000 curr count: 35147
2021/12/09 08:54:28 Waiting for src bucket item count to get to: 50000 curr count: 35147
2021/12/09 08:54:33 Waiting for src bucket item count to get to: 50000 curr count: 45022
2021/12/09 08:54:38 Waiting for src bucket item count to get to: 50000 curr count: 45022
2021/12/09 08:54:43 Waiting for src bucket item count to get to: 50000 curr count: 45022
2021/12/09 08:54:53 Waiting for dst bucket item count to get to: 50000 curr count: 31105
2021/12/09 08:54:58 Waiting for dst bucket item count to get to: 50000 curr count: 31105
2021/12/09 08:55:03 Waiting for dst bucket item count to get to: 50000 curr count: 41171
2021/12/09 08:55:08 Waiting for dst bucket item count to get to: 50000 curr count: 41171
2021/12/09 08:55:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:55:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:55:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:55:13 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:55:18 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:55:23 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:55:28 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:55:33 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:55:38 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/09 08:55:38 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/09 08:55:38 No eventing-con process running
2021/12/09 08:55:38 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/09 08:55:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:55:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:55:50 Waiting for dst bucket item count to get to: 0 curr count: 16311
2021/12/09 08:55:55 Waiting for dst bucket item count to get to: 0 curr count: 16311
2021/12/09 08:55:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:55:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:56:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:56:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.22s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/09 08:56:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:56:17 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/09 08:56:17 No eventing-con process running
2021/12/09 08:56:17 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/09 08:56:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:56:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:56:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:56:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:56:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:56:36 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:56:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 08:56:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 08:56:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 08:56:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 08:56:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/09 08:56:57 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/09 08:56:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 08:56:57 Starting bucket ops to source bucket
2021/12/09 08:56:58 Removing node(s): [] from the cluster
2021/12/09 08:57:03 Rebalance progress: 12.5
2021/12/09 08:57:08 Rebalance progress: 12.5
2021/12/09 08:57:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:57:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:57:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:57:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 08:57:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 08:57:18 Rebalance progress: 12.5
2021/12/09 08:57:23 Rebalance progress: 12.5
2021/12/09 08:57:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:57:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 08:57:33 Rebalance progress: 12.5
2021/12/09 08:57:38 Rebalance progress: 12.5
2021/12/09 08:57:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:57:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:57:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 08:57:43 Waiting for src bucket item count to get to: 50000 curr count: 49892
2021/12/09 08:57:48 Waiting for src bucket item count to get to: 50000 curr count: 49892
2021/12/09 08:57:58 Waiting for dst bucket item count to get to: 50000 curr count: 35297
2021/12/09 08:58:03 Waiting for dst bucket item count to get to: 50000 curr count: 35297
2021/12/09 08:58:08 Waiting for dst bucket item count to get to: 50000 curr count: 42974
2021/12/09 08:58:13 Waiting for dst bucket item count to get to: 50000 curr count: 42974
2021/12/09 08:58:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 08:58:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 08:58:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 08:58:18 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:58:23 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:58:28 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:58:34 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:58:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:58:44 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/09 08:58:44 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/09 08:58:44 No eventing-con process running
2021/12/09 08:58:44 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/09 08:58:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:58:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:58:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:58:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:59:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:59:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 08:59:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.11s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/09 08:59:17 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 08:59:17 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/09 08:59:17 No eventing-con process running
2021/12/09 08:59:17 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/09 08:59:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:59:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:59:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 08:59:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:59:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:59:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 08:59:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 08:59:36 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 08:59:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 08:59:42 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 08:59:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 08:59:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 08:59:57 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/09 08:59:57 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/09 08:59:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 08:59:57 Starting bucket ops to source bucket
2021/12/09 08:59:57 Removing node(s): [] from the cluster
2021/12/09 09:00:02 Rebalance progress: 12.5
2021/12/09 09:00:07 Rebalance progress: 12.5
2021/12/09 09:00:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:00:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:00:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:00:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:00:17 Rebalance progress: 12.5
2021/12/09 09:00:22 Rebalance progress: 12.5
2021/12/09 09:00:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:00:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:00:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:00:27 Waiting for src bucket item count to get to: 50000 curr count: 28948
2021/12/09 09:00:32 Waiting for src bucket item count to get to: 50000 curr count: 39864
2021/12/09 09:00:37 Waiting for src bucket item count to get to: 50000 curr count: 39864
2021/12/09 09:00:47 Waiting for dst bucket item count to get to: 50000 curr count: 26531
2021/12/09 09:00:52 Waiting for dst bucket item count to get to: 50000 curr count: 26531
2021/12/09 09:00:57 Waiting for dst bucket item count to get to: 50000 curr count: 35834
2021/12/09 09:01:02 Waiting for dst bucket item count to get to: 50000 curr count: 35834
2021/12/09 09:01:07 Waiting for dst bucket item count to get to: 50000 curr count: 44102
2021/12/09 09:01:12 Waiting for dst bucket item count to get to: 50000 curr count: 44102
2021/12/09 09:01:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:01:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:01:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:01:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 09:01:23 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 09:01:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 09:01:33 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 09:01:38 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 09:01:43 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/09 09:01:43 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/09 09:01:43 No eventing-con process running
2021/12/09 09:01:43 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/09 09:01:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:01:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:01:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:01:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:02:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:02:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.23s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/09 09:02:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:02:12 Removing node(s): [] from the cluster
2021/12/09 09:02:17 Rebalance progress: 100
2021/12/09 09:02:22 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:02:22 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/09 09:02:22 No eventing-con process running
2021/12/09 09:02:22 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/09 09:02:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:02:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:02:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:02:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:02:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:02:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:02:40 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:02:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 09:02:46 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 09:02:51 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 09:02:56 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 09:03:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/09 09:03:01 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/09 09:03:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:03:01 Starting bucket ops to source bucket
2021/12/09 09:03:07 Rebalance progress: 12.5
2021/12/09 09:03:12 Rebalance progress: 12.5
2021/12/09 09:03:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:03:17 Waiting for src bucket item count to get to: 50000 curr count: 10019
2021/12/09 09:03:22 Waiting for src bucket item count to get to: 50000 curr count: 23623
2021/12/09 09:03:27 Waiting for src bucket item count to get to: 50000 curr count: 23623
2021/12/09 09:03:32 Waiting for src bucket item count to get to: 50000 curr count: 37556
2021/12/09 09:03:37 Waiting for src bucket item count to get to: 50000 curr count: 37556
2021/12/09 09:03:42 Waiting for src bucket item count to get to: 50000 curr count: 37556
2021/12/09 09:03:47 Waiting for src bucket item count to get to: 50000 curr count: 46973
2021/12/09 09:03:52 Waiting for src bucket item count to get to: 50000 curr count: 46973
2021/12/09 09:04:02 Waiting for dst bucket item count to get to: 50000 curr count: 28107
2021/12/09 09:04:07 Waiting for dst bucket item count to get to: 50000 curr count: 28107
2021/12/09 09:04:12 Waiting for dst bucket item count to get to: 50000 curr count: 44449
2021/12/09 09:04:17 Waiting for dst bucket item count to get to: 50000 curr count: 44449
2021/12/09 09:04:17 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:04:17 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:04:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:04:22 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:04:27 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:04:32 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:04:37 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:04:42 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:04:47 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/09 09:04:47 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/09 09:04:47 No eventing-con process running
2021/12/09 09:04:47 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/09 09:04:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:04:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:04:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:05:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:05:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:05:11 Waiting for dst bucket item count to get to: 0 curr count: 34550
2021/12/09 09:05:16 Waiting for dst bucket item count to get to: 0 curr count: 34550
2021/12/09 09:05:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.27s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/09 09:05:26 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:05:26 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/09 09:05:26 No eventing-con process running
2021/12/09 09:05:26 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/09 09:05:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:05:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:05:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:05:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:05:44 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:05:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 09:05:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 09:05:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 09:06:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 09:06:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/09 09:06:05 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/09 09:06:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:06:05 Starting bucket ops to source bucket
2021/12/09 09:06:06 Removing node(s): [] from the cluster
2021/12/09 09:06:11 Rebalance progress: 12.5
2021/12/09 09:06:16 Rebalance progress: 12.5
2021/12/09 09:06:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:06:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:06:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:06:26 Rebalance progress: 12.5
2021/12/09 09:06:31 Rebalance progress: 12.5
2021/12/09 09:06:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:06:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:06:41 Rebalance progress: 12.5
2021/12/09 09:06:46 Rebalance progress: 12.5
2021/12/09 09:06:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:06:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:06:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:06:51 Waiting for src bucket item count to get to: 50000 curr count: 49900
2021/12/09 09:06:56 Waiting for src bucket item count to get to: 50000 curr count: 49900
2021/12/09 09:07:06 Waiting for dst bucket item count to get to: 50000 curr count: 38925
2021/12/09 09:07:11 Waiting for dst bucket item count to get to: 50000 curr count: 38925
2021/12/09 09:07:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:07:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:07:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:07:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:07:21 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:07:26 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:07:31 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:07:36 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:07:41 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/09 09:07:41 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/09 09:07:41 No eventing-con process running
2021/12/09 09:07:41 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/09 09:07:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:07:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:07:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:07:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:07:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:08:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:08:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:08:10 Waiting for dst bucket item count to get to: 0 curr count: 6911
2021/12/09 09:08:15 Waiting for dst bucket item count to get to: 0 curr count: 6911
2021/12/09 09:08:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.40s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/09 09:08:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 09:08:25 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/09 09:08:25 No eventing-con process running
2021/12/09 09:08:25 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/09 09:08:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:08:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:08:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:08:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:08:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:08:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:08:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:08:44 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:08:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 09:08:50 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 09:08:55 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 09:09:00 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 09:09:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/09 09:09:05 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/09 09:09:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 09:09:05 Starting bucket ops to source bucket
2021/12/09 09:09:05 Removing node(s): [] from the cluster
2021/12/09 09:09:10 Rebalance progress: 12.5
2021/12/09 09:09:15 Rebalance progress: 12.5
2021/12/09 09:09:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:09:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:09:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:09:25 Rebalance progress: 12.5
2021/12/09 09:09:30 Rebalance progress: 12.5
2021/12/09 09:09:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:09:35 Waiting for src bucket item count to get to: 50000 curr count: 29840
2021/12/09 09:09:40 Waiting for src bucket item count to get to: 50000 curr count: 41709
2021/12/09 09:09:45 Waiting for src bucket item count to get to: 50000 curr count: 41709
2021/12/09 09:09:55 Waiting for dst bucket item count to get to: 50000 curr count: 33813
2021/12/09 09:10:00 Waiting for dst bucket item count to get to: 50000 curr count: 33813
2021/12/09 09:10:05 Waiting for dst bucket item count to get to: 50000 curr count: 43256
2021/12/09 09:10:10 Waiting for dst bucket item count to get to: 50000 curr count: 43256
2021/12/09 09:10:10 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:10:10 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:10:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:10:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 09:10:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 09:10:26 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 09:10:31 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 09:10:36 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/09 09:10:36 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/09 09:10:36 No eventing-con process running
2021/12/09 09:10:36 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/09 09:10:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:10:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:10:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:10:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:10:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:10:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:10:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (164.23s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/09 09:11:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 09:11:05 Removing node(s): [] from the cluster
2021/12/09 09:11:10 Rebalance progress: 100
2021/12/09 09:11:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:11:15 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/09 09:11:15 No eventing-con process running
2021/12/09 09:11:15 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/09 09:11:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:11:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:11:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:11:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:11:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:11:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:11:33 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:11:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 09:11:39 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 09:11:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 09:11:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 09:11:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/09 09:11:54 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/09 09:11:54 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:11:54 Starting bucket ops to source bucket
2021/12/09 09:11:59 Rebalance progress: 12.5
2021/12/09 09:12:04 Rebalance progress: 12.5
2021/12/09 09:12:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:12:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:12:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:12:09 Waiting for src bucket item count to get to: 50000 curr count: 6507
2021/12/09 09:12:14 Waiting for src bucket item count to get to: 50000 curr count: 19967
2021/12/09 09:12:19 Waiting for src bucket item count to get to: 50000 curr count: 19967
2021/12/09 09:12:24 Waiting for src bucket item count to get to: 50000 curr count: 30226
2021/12/09 09:12:30 Waiting for src bucket item count to get to: 50000 curr count: 30226
2021/12/09 09:12:35 Waiting for src bucket item count to get to: 50000 curr count: 30226
2021/12/09 09:12:40 Waiting for src bucket item count to get to: 50000 curr count: 42505
2021/12/09 09:12:50 Waiting for dst bucket item count to get to: 50000 curr count: 41322
2021/12/09 09:12:55 Waiting for dst bucket item count to get to: 50000 curr count: 41322
2021/12/09 09:13:00 Waiting for dst bucket item count to get to: 50000 curr count: 48569
2021/12/09 09:13:05 Waiting for dst bucket item count to get to: 50000 curr count: 48569
2021/12/09 09:13:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:13:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:13:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:13:10 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:13:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:13:20 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:13:25 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:13:30 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:13:35 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/09 09:13:35 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/09 09:13:35 No eventing-con process running
2021/12/09 09:13:35 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/09 09:13:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:13:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:13:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:13:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:13:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:13:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:13:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:13:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.24s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/09 09:14:08 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:14:08 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/09 09:14:09 No eventing-con process running
2021/12/09 09:14:09 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/09 09:14:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:14:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:14:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:14:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:14:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:14:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:14:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:14:27 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:14:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 09:14:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 09:14:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 09:14:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 09:14:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/09 09:14:48 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/09 09:14:48 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 09:14:48 Starting bucket ops to source bucket
2021/12/09 09:14:49 Removing node(s): [] from the cluster
2021/12/09 09:14:54 Rebalance progress: 12.5
2021/12/09 09:14:59 Rebalance progress: 12.5
2021/12/09 09:15:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:15:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:15:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 09:15:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:15:09 Rebalance progress: 12.5
2021/12/09 09:15:14 Rebalance progress: 12.5
2021/12/09 09:15:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:15:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:15:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:15:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:15:24 Rebalance progress: 12.5
2021/12/09 09:15:29 Rebalance progress: 12.5
2021/12/09 09:15:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:15:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:15:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:15:34 Waiting for src bucket item count to get to: 50000 curr count: 48584
2021/12/09 09:15:39 Waiting for src bucket item count to get to: 50000 curr count: 48584
2021/12/09 09:15:49 Waiting for dst bucket item count to get to: 50000 curr count: 40297
2021/12/09 09:15:54 Waiting for dst bucket item count to get to: 50000 curr count: 40297
2021/12/09 09:15:59 Waiting for dst bucket item count to get to: 50000 curr count: 48823
2021/12/09 09:16:04 Waiting for dst bucket item count to get to: 50000 curr count: 48823
2021/12/09 09:16:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:16:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:16:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:16:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:16:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:16:20 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:16:25 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:16:30 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:16:35 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/09 09:16:35 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/09 09:16:35 No eventing-con process running
2021/12/09 09:16:35 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/09 09:16:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:16:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:16:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:16:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:16:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:16:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:16:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:16:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.71s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/09 09:17:08 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:17:08 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/09 09:17:08 No eventing-con process running
2021/12/09 09:17:08 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/09 09:17:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:17:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:17:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:17:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:17:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:17:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:17:27 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:17:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 09:17:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 09:17:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 09:17:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 09:17:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/09 09:17:48 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/09 09:17:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:17:48 Starting bucket ops to source bucket
2021/12/09 09:17:48 Removing node(s): [] from the cluster
2021/12/09 09:17:53 Rebalance progress: 12.5
2021/12/09 09:17:58 Rebalance progress: 12.5
2021/12/09 09:18:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:18:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:18:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:18:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:18:08 Rebalance progress: 12.5
2021/12/09 09:18:13 Rebalance progress: 12.5
2021/12/09 09:18:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:18:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:18:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:18:18 Waiting for src bucket item count to get to: 50000 curr count: 26904
2021/12/09 09:18:23 Waiting for src bucket item count to get to: 50000 curr count: 36951
2021/12/09 09:18:28 Waiting for src bucket item count to get to: 50000 curr count: 36951
2021/12/09 09:18:33 Waiting for src bucket item count to get to: 50000 curr count: 49731
2021/12/09 09:18:38 Waiting for src bucket item count to get to: 50000 curr count: 49731
2021/12/09 09:18:49 Waiting for dst bucket item count to get to: 50000 curr count: 45608
2021/12/09 09:18:54 Waiting for dst bucket item count to get to: 50000 curr count: 45608
2021/12/09 09:18:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:18:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:18:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:18:59 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:19:04 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:19:09 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:19:14 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:19:19 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:19:24 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/09 09:19:24 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/09 09:19:24 No eventing-con process running
2021/12/09 09:19:24 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/09 09:19:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:19:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:19:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:19:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:19:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:19:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:19:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:19:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.26s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/09 09:19:52 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:19:53 Removing node(s): [] from the cluster
2021/12/09 09:19:58 Rebalance progress: 100
2021/12/09 09:20:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:20:03 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/09 09:20:03 No eventing-con process running
2021/12/09 09:20:03 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/09 09:20:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:20:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:20:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:20:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:20:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:20:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:20:21 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:20:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 09:20:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 09:20:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 09:20:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 09:20:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/09 09:20:42 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/09 09:20:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:20:42 Starting bucket ops to source bucket
2021/12/09 09:20:48 Rebalance progress: 12.5
2021/12/09 09:20:53 Rebalance progress: 12.5
2021/12/09 09:20:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:20:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:20:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:20:58 Waiting for src bucket item count to get to: 50000 curr count: 9279
2021/12/09 09:21:03 Waiting for src bucket item count to get to: 50000 curr count: 22713
2021/12/09 09:21:08 Waiting for src bucket item count to get to: 50000 curr count: 22713
2021/12/09 09:21:13 Waiting for src bucket item count to get to: 50000 curr count: 34598
2021/12/09 09:21:18 Waiting for src bucket item count to get to: 50000 curr count: 34598
2021/12/09 09:21:23 Waiting for src bucket item count to get to: 50000 curr count: 44688
2021/12/09 09:21:28 Waiting for src bucket item count to get to: 50000 curr count: 44688
2021/12/09 09:21:33 Waiting for src bucket item count to get to: 50000 curr count: 44688
2021/12/09 09:21:43 Waiting for dst bucket item count to get to: 50000 curr count: 39585
2021/12/09 09:21:48 Waiting for dst bucket item count to get to: 50000 curr count: 39585
2021/12/09 09:21:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:21:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:21:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:21:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:21:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:22:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:22:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:22:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:22:18 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/09 09:22:18 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/09 09:22:18 No eventing-con process running
2021/12/09 09:22:18 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/09 09:22:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:22:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:22:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:22:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:22:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:22:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.17s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/09 09:22:47 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:22:47 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/09 09:22:47 No eventing-con process running
2021/12/09 09:22:47 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/09 09:22:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:22:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:22:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:22:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:23:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:23:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:23:05 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:23:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 09:23:11 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 09:23:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 09:23:21 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 09:23:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/09 09:23:26 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/09 09:23:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:23:26 Starting bucket ops to source bucket
2021/12/09 09:23:27 Removing node(s): [] from the cluster
2021/12/09 09:23:32 Rebalance progress: 12.5
2021/12/09 09:23:37 Rebalance progress: 12.5
2021/12/09 09:23:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:23:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:23:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:23:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:23:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:23:48 Rebalance progress: 12.5
2021/12/09 09:23:53 Rebalance progress: 12.5
2021/12/09 09:23:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:23:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:23:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:23:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:24:03 Rebalance progress: 12.5
2021/12/09 09:24:08 Rebalance progress: 12.5
2021/12/09 09:24:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:24:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:24:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:24:13 Waiting for src bucket item count to get to: 50000 curr count: 49980
2021/12/09 09:24:18 Waiting for src bucket item count to get to: 50000 curr count: 49980
2021/12/09 09:24:28 Waiting for dst bucket item count to get to: 50000 curr count: 40466
2021/12/09 09:24:33 Waiting for dst bucket item count to get to: 50000 curr count: 40466
2021/12/09 09:24:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/09 09:24:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/09 09:24:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:24:38 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:24:43 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:24:48 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:24:53 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:24:58 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:25:03 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/09 09:25:03 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/09 09:25:03 No eventing-con process running
2021/12/09 09:25:03 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/09 09:25:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:25:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:25:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:25:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:25:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:25:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:25:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/09 09:25:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.28s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/09 09:25:37 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:25:37 App: bucket_op_on_update got un-deployed
2021/12/09 09:25:37 No eventing-con process running
2021/12/09 09:25:37 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:25:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:25:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:25:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:25:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:25:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:25:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:25:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:25:55 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:25:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:26:02 Starting bucket ops to source bucket
2021/12/09 09:26:07 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:26:12 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:26:17 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:26:17 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:26:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:26:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:26:22 Rebalance progress: 12.5
2021/12/09 09:26:27 Rebalance progress: 12.5
2021/12/09 09:26:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:26:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:26:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:26:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:26:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:26:38 Rebalance progress: 12.5
2021/12/09 09:26:43 Rebalance progress: 12.5
2021/12/09 09:26:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:26:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 09:26:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:26:53 Rebalance progress: 12.5
2021/12/09 09:26:58 Rebalance progress: 12.5
2021/12/09 09:27:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:27:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:27:08 Rebalance progress: 12.5
2021/12/09 09:27:13 Rebalance progress: 12.5
2021/12/09 09:27:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:27:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:27:23 Rebalance progress: 12.5
2021/12/09 09:27:28 Rebalance progress: 12.5
2021/12/09 09:27:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:27:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 09:27:38 Rebalance progress: 12.5
2021/12/09 09:27:43 Rebalance progress: 12.5
2021/12/09 09:27:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:27:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:27:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:27:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:28:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:28:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:28:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:28:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:28:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:28:19 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:28:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:28:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:28:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:28:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6226
2021/12/09 09:28:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6226
2021/12/09 09:28:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:28:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:28:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:28:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10673
2021/12/09 09:28:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.41s)
=== RUN   TestKVForceFailover
2021/12/09 09:29:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:29:02 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:29:02 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:29:02 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:29:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:29:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:29:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:29:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:29:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:29:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:29:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:29:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:29:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:29:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:29:22 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:29:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:29:23 Starting bucket ops to source bucket
2021/12/09 09:29:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:29:33 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:29:38 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:29:43 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:29:43 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:29:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:29:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:29:43 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/09 09:30:03 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/09 09:30:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 09:30:08 Rebalance progress: 12.5
2021/12/09 09:30:13 Rebalance progress: 12.5
2021/12/09 09:30:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:30:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 09:30:24 Rebalance progress: 12.5
2021/12/09 09:30:29 Rebalance progress: 12.5
2021/12/09 09:30:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:30:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:30:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:30:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:30:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:30:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:30:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:31:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:31:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:31:04 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:31:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:31:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:31:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:31:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7081
2021/12/09 09:31:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:31:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:31:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:31:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7081
2021/12/09 09:31:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (160.07s)
=== RUN   TestKVHardFailoverRecovery
2021/12/09 09:31:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:31:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:31:37 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:31:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:31:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:31:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:31:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:31:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:31:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:31:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:31:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:31:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:31:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:32:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:32:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:32:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:32:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:32:17 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:32:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:32:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:32:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:32:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:32:23 Rebalance progress: 12.5
2021/12/09 09:32:28 Rebalance progress: 12.5
2021/12/09 09:32:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:32:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:32:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:32:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/09 09:33:33 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/09 09:33:38 Rebalance progress: 12.5
2021/12/09 09:33:43 Rebalance progress: 12.5
2021/12/09 09:33:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:33:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:33:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:33:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:33:53 Rebalance progress: 12.5
2021/12/09 09:33:58 Rebalance progress: 12.5
2021/12/09 09:34:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:34:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:34:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:34:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:34:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:34:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:34:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:34:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:34:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:34:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:34:28 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:34:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:34:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:34:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:34:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2379
2021/12/09 09:34:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2379
2021/12/09 09:34:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:34:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:34:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:34:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12070
2021/12/09 09:34:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.48s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/09 09:35:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:35:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:35:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:35:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:35:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:35:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:35:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:35:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:35:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:35:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:35:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:35:25 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:35:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:35:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:35:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:35:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:35:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:35:47 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:35:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:35:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:35:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:35:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:35:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 09:35:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:35:53 Rebalance progress: 12.5
2021/12/09 09:35:58 Rebalance progress: 12.5
2021/12/09 09:36:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:36:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 09:36:08 Rebalance progress: 12.5
2021/12/09 09:36:13 Rebalance progress: 12.5
2021/12/09 09:36:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:36:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:36:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:36:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:36:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:36:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:36:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:36:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:36:43 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:36:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:36:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:36:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:36:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4618
2021/12/09 09:36:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:36:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:37:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:37:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4618
2021/12/09 09:37:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.90s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/09 09:37:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:37:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:37:17 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:37:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:37:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:37:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:37:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:37:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:37:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:37:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:37:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:37:31 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:37:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:37:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:37:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:37:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:37:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:37:52 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:37:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:37:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:37:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:37:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:37:58 Rebalance progress: 12.5
2021/12/09 09:38:03 Rebalance progress: 12.5
2021/12/09 09:38:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:38:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:38:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:38:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:38:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:38:13 Rebalance progress: 12.5
2021/12/09 09:38:18 Rebalance progress: 12.5
2021/12/09 09:38:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:38:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:38:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:38:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:38:28 Rebalance progress: 12.5
2021/12/09 09:38:33 Rebalance progress: 12.5
2021/12/09 09:38:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:38:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:38:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:38:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:38:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:38:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:38:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:38:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:39:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:39:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:39:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:39:08 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:39:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:39:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:39:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:39:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1990
2021/12/09 09:39:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1990
2021/12/09 09:39:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:39:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:39:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:39:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6116
2021/12/09 09:39:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:39:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:39:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:39:42 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:39:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:39:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:39:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:39:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:39:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:39:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.79s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/09 09:40:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:40:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:40:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:40:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:40:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:40:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:40:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:40:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:40:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:40:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:40:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:40:24 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 09:40:24 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:40:24 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/09 09:40:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:40:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:40:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:40:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:40:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:40:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:40:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:40:38 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:40:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:40:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:40:39 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:40:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 09:40:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:40:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:40:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:41:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:41:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/09 09:41:05 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 09:41:10 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 09:41:15 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 09:41:15 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:41:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:41:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:41:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:41:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:41:20 Rebalance progress: 12.5
2021/12/09 09:41:25 Rebalance progress: 12.5
2021/12/09 09:41:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:41:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:41:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:41:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:41:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:41:36 Rebalance progress: 12.5
2021/12/09 09:41:41 Rebalance progress: 12.5
2021/12/09 09:41:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:41:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:41:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:41:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 09:41:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:41:51 Rebalance progress: 12.5
2021/12/09 09:41:56 Rebalance progress: 12.5
2021/12/09 09:42:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:42:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:42:06 Rebalance progress: 12.5
2021/12/09 09:42:11 Rebalance progress: 12.5
2021/12/09 09:42:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:42:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:42:22 Rebalance progress: 12.5
2021/12/09 09:42:27 Rebalance progress: 12.5
2021/12/09 09:42:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:42:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 09:42:37 Rebalance progress: 12.5
2021/12/09 09:42:42 Rebalance progress: 12.5
2021/12/09 09:42:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:42:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:42:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:42:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:42:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:43:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:43:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:43:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:43:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:43:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:43:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:43:22 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:43:22 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:43:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:43:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:43:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12170
2021/12/09 09:43:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:43:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:43:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:43:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12170
2021/12/09 09:43:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:43:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:43:52 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:43:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:44:02 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:44:07 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:44:13 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:44:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:44:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 09:44:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:44:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/09 09:44:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:44:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:44:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:44:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:44:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:44:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.53s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/09 09:44:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:44:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:44:41 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:44:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:44:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:44:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:44:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:44:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:44:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:44:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:45:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:45:00 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 09:45:00 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:45:00 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/09 09:45:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:45:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:45:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:45:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:45:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:45:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:45:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:45:13 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:45:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:45:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:45:14 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:45:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 09:45:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:45:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:45:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:45:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:45:35 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/09 09:45:40 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 09:45:45 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 09:45:50 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/09 09:45:50 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:45:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:45:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:45:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:45:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:45:56 Rebalance progress: 12.5
2021/12/09 09:46:01 Rebalance progress: 12.5
2021/12/09 09:46:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:46:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:46:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:46:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:46:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:46:11 Rebalance progress: 12.5
2021/12/09 09:46:16 Rebalance progress: 12.5
2021/12/09 09:46:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:46:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:46:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:46:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:46:26 Rebalance progress: 12.5
2021/12/09 09:46:31 Rebalance progress: 12.5
2021/12/09 09:46:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:46:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:46:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:46:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:46:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:46:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:47:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:47:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:47:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:47:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:47:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:47:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:47:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:47:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:47:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:47:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:47:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7602
2021/12/09 09:47:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:47:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:47:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:47:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:47:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:47:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:47:52 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:47:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/09 09:47:57 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/09 09:47:57 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:47:57 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/09 09:47:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:48:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:48:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:48:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:48:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:48:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (219.83s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 09:48:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:48:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:48:21 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:48:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:48:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:48:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:48:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:48:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:48:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:48:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:48:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:48:39 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:48:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:48:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:48:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:48:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:49:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:49:01 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:49:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:49:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:49:21 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:49:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:49:37 Rebalance progress: 12.5
2021/12/09 09:49:42 Rebalance progress: 12.5
2021/12/09 09:49:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:49:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:49:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:49:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:49:52 Rebalance progress: 12.5
2021/12/09 09:49:57 Rebalance progress: 12.5
2021/12/09 09:50:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:50:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:50:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:50:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:50:07 Rebalance progress: 12.5
2021/12/09 09:50:12 Rebalance progress: 12.5
2021/12/09 09:50:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:50:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:50:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:50:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:50:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:50:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:50:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:50:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:50:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:50:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:50:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:50:48 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:50:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:50:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:50:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:50:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:50:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:51:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9188
2021/12/09 09:51:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.40s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/09 09:51:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:51:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:51:16 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:51:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/09 09:51:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:51:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:51:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:51:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:51:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:51:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:51:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:51:35 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:51:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:51:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 09:51:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:51:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:51:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:51:56 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:51:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:51:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:51:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:51:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:52:02 Rebalance progress: 12.5
2021/12/09 09:52:07 Rebalance progress: 12.5
2021/12/09 09:52:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:52:12 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:52:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:52:27 Rebalance progress: 12.5
2021/12/09 09:52:32 Rebalance progress: 12.5
2021/12/09 09:52:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:52:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 09:52:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 09:52:42 Rebalance progress: 12.5
2021/12/09 09:52:47 Rebalance progress: 12.5
2021/12/09 09:52:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:52:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 09:52:57 Rebalance progress: 12.5
2021/12/09 09:53:02 Rebalance progress: 12.5
2021/12/09 09:53:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:53:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:53:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:53:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:53:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:53:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:53:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:53:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:53:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:53:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:53:32 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:53:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:53:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:53:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:53:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8592
2021/12/09 09:53:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8592
2021/12/09 09:53:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:53:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:53:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:54:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7546
2021/12/09 09:54:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7546
2021/12/09 09:54:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.54s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/09 09:54:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 09:54:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 09:54:16 Rebalance progress: 100
2021/12/09 09:54:16 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:54:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:54:46 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 09:54:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 09:54:52 Removing node(s): [] from the cluster
2021/12/09 09:54:52 Starting bucket ops to source bucket
2021/12/09 09:54:58 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/09 09:55:03 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:55:08 Waiting for app: bucket_op_on_update to get deployed
2021/12/09 09:55:08 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:55:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:55:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 09:55:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 09:55:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:55:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:55:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:55:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:55:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/09 09:55:33 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/09 09:55:33 eventingSetup.go:46: No eventing-con process running
2021/12/09 09:55:33 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/09 09:55:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:55:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:55:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/09 09:55:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 09:55:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 09:55:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 09:55:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/09 09:55:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.65s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	11998.185s

Rebalance tests took 3 hours, 20 minutes and 1 seconds.

Testing Succeeded


Version: versions-09.12.2021-02.16.cfg
Build Log: make-09.12.2021-02.16.log
Server Log: logs-09.12.2021-02.16.tar.gz

Finished