Building

Started building at 2021/12/14 18:35:11
Calculating base
Updating mirror
Basing run on 7.1.0-1908 07491e50b
Updating tree for run 14.12.2021-18.35
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 8cc3b2c, changes since last good build: none
kv_engine is at fb4466d, changes since last good build: 
 fb4466db3 MB-49121: Remove ThreadGate from hdr_histogram tests
 6244f35c7 Clang-Tidy: ep_engine.cc
 a70581c5a Fix log message
 984957b00 MB-50060: Fail compaction if prepare CompactKVStore fails
ns_server is at abe4a02, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 45ee69a, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable afc8b137] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 48c807eb] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 0812c6cb] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 104d98df] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 01a833e4] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/6
Total 24 (delta 20), reused 24 (delta 20)
[unstable 06f5cbc4] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/9
Total 56 (delta 47), reused 56 (delta 47)
[unstable 40f1a41a] MB-46304: Add functionality to disable curl feature
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 193 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/7
Total 67 (delta 57), reused 67 (delta 57)
[unstable 016a993f] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 149 insertions(+), 74 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/4
Total 82 (delta 64), reused 74 (delta 64)
[unstable 17ed56cb] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/2
Total 90 (delta 69), reused 80 (delta 69)
[unstable 2348c8d5] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/3
Total 5 (delta 2), reused 5 (delta 2)
[unstable 6a727b6] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/14 18:52:46
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/14 18:54:18 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:51562->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:51562->127.0.0.1:9000: read: connection reset by peer
2021/12/14 18:54:28 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 18:54:34 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/14 18:54:34 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 18:54:39 Removing node(s): [] from the cluster
2021/12/14 18:54:44 Rebalance progress: 0
2021/12/14 18:54:49 Rebalance progress: 0.2531828703703704
2021/12/14 18:54:54 Rebalance progress: 4.937065972222222
2021/12/14 18:54:59 Rebalance progress: 9.25925925925926
2021/12/14 18:55:04 Rebalance progress: 13.00274884259259
2021/12/14 18:55:09 Rebalance progress: 17.72280092592592
2021/12/14 18:55:14 Rebalance progress: 21.03226273148148
2021/12/14 18:55:19 Rebalance progress: 25.64380787037037
2021/12/14 18:55:24 Rebalance progress: 100
2021/12/14 18:55:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:55:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:55:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:56:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:56:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:56:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:56:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:57:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:57:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:57:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 18:57:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/14 18:58:15 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:58:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/14 18:58:21 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 18:58:26 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 18:58:31 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 18:58:36 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 18:58:36 App: TestAdvancedGetOps got deployed
2021/12/14 18:58:36 Starting bucket ops to source bucket
2021/12/14 18:58:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 18:58:46 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 18:58:46 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 18:58:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 18:58:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 18:58:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:58:51 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 18:58:56 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 18:59:01 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 18:59:06 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 18:59:06 App: TestAdvancedGetOps got un-deployed
2021/12/14 18:59:06 No eventing-con process running
2021/12/14 18:59:06 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/14 18:59:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 18:59:18 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/14 18:59:23 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/14 18:59:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:59:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 18:59:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 18:59:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (80.10s)
=== RUN   TestAdvancedInsertOps
2021/12/14 18:59:35 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:59:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 18:59:41 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 18:59:46 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 18:59:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 18:59:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 18:59:56 App: TestAdvancedInsertOps got deployed
2021/12/14 18:59:56 Starting bucket ops to source bucket
2021/12/14 19:00:02 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 19:00:02 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 19:00:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:00:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:00:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:00:07 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:00:12 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:00:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:00:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:00:22 App: TestAdvancedInsertOps got un-deployed
2021/12/14 19:00:22 No eventing-con process running
2021/12/14 19:00:22 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/14 19:00:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:00:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:00:39 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 19:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:00:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:00:50 Waiting for dst bucket item count to get to: 0 curr count: 817
2021/12/14 19:00:55 Waiting for dst bucket item count to get to: 0 curr count: 817
2021/12/14 19:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:00:55 Testing insert operation on source bucket
2021/12/14 19:00:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:00:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:01:02 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 19:01:07 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 19:01:12 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 19:01:17 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 19:01:17 App: TestAdvancedInsertOps got deployed
2021/12/14 19:01:17 Starting bucket ops to source bucket
2021/12/14 19:01:23 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 19:01:28 Waiting for dst bucket item count to get to: 2048 curr count: 1616
2021/12/14 19:01:33 Waiting for dst bucket item count to get to: 2048 curr count: 1616
2021/12/14 19:01:33 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 19:01:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:01:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:01:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:01:38 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:01:43 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:01:48 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:01:53 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 19:01:53 App: TestAdvancedInsertOps got un-deployed
2021/12/14 19:01:53 No eventing-con process running
2021/12/14 19:01:53 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/14 19:01:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:02:04 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 19:02:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:02:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:02:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.55s)
=== RUN   TestAdvancedUpsertOps
2021/12/14 19:02:11 Starting bucket ops to source bucket
2021/12/14 19:02:12 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:02:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:02:18 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:02:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:02:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:02:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:02:33 App: TestAdvancedUpsertOps got deployed
2021/12/14 19:02:33 Starting bucket ops to source bucket
2021/12/14 19:02:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:02:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:02:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:02:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:02:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:02:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:02:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:02:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:02:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:03:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:03:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/14 19:03:04 No eventing-con process running
2021/12/14 19:03:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/14 19:03:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:03:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:03:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:03:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:03:23 Testing upsert operation on source bucket
2021/12/14 19:03:23 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:03:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:03:29 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:03:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:03:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:03:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 19:03:44 App: TestAdvancedUpsertOps got deployed
2021/12/14 19:03:44 Starting bucket ops to source bucket
2021/12/14 19:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:03:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:03:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:03:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:03:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:03:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:04:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:04:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 19:04:05 App: TestAdvancedUpsertOps got un-deployed
2021/12/14 19:04:05 No eventing-con process running
2021/12/14 19:04:05 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/14 19:04:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:04:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:04:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (127.94s)
=== RUN   TestAdvancedReplaceOps
2021/12/14 19:04:19 Starting bucket ops to source bucket
2021/12/14 19:04:20 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:04:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:04:26 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:04:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:04:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:04:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:04:41 App: TestAdvancedReplaceOps got deployed
2021/12/14 19:04:41 Starting bucket ops to source bucket
2021/12/14 19:04:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:04:52 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:04:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:04:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:04:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:04:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:04:58 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:05:03 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:05:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:05:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:05:13 App: TestAdvancedReplaceOps got un-deployed
2021/12/14 19:05:13 No eventing-con process running
2021/12/14 19:05:13 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/14 19:05:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:05:24 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:05:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:05:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:05:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:05:31 Testing upsert operation on source bucket
2021/12/14 19:05:31 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:05:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:05:37 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:05:42 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:05:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:05:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 19:05:52 App: TestAdvancedReplaceOps got deployed
2021/12/14 19:05:52 Starting bucket ops to source bucket
2021/12/14 19:05:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:05:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:05:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:05:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:05:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:06:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:06:09 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:06:14 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 19:06:14 App: TestAdvancedReplaceOps got un-deployed
2021/12/14 19:06:14 No eventing-con process running
2021/12/14 19:06:14 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/14 19:06:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:06:25 Waiting for dst bucket item count to get to: 0 curr count: 170
2021/12/14 19:06:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:06:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:06:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.95s)
=== RUN   TestAdvancedDeleteOps
2021/12/14 19:06:32 Starting bucket ops to source bucket
2021/12/14 19:06:32 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:06:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:06:38 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:06:43 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:06:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:06:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:06:53 App: TestAdvancedDeleteOps got deployed
2021/12/14 19:06:53 Starting bucket ops to source bucket
2021/12/14 19:06:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:07:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:07:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:07:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:07:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:07:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:07:09 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:07:14 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:07:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:07:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:07:24 App: TestAdvancedDeleteOps got un-deployed
2021/12/14 19:07:24 No eventing-con process running
2021/12/14 19:07:24 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/14 19:07:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:07:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:07:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:07:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:07:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:07:42 Testing delete operation on source bucket
2021/12/14 19:07:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:07:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/14 19:07:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:07:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:07:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:08:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 19:08:03 App: TestAdvancedDeleteOps got deployed
2021/12/14 19:08:03 Starting bucket ops to source bucket
2021/12/14 19:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:08:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:08:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:08:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:08:09 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:08:14 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:08:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:08:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 19:08:24 App: TestAdvancedDeleteOps got un-deployed
2021/12/14 19:08:24 No eventing-con process running
2021/12/14 19:08:24 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/14 19:08:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:08:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:08:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:08:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.58s)
=== RUN   TestEnoentAdvancedGet
2021/12/14 19:08:38 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:08:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:08:44 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:08:49 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:08:54 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:08:59 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:08:59 App: TestEnoentAdvancedGet got deployed
2021/12/14 19:08:59 Starting bucket ops to source bucket
2021/12/14 19:09:04 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:09:09 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:09:09 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:09:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:09:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:09:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:09:14 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:09:19 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:09:24 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:09:29 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:09:29 App: TestEnoentAdvancedGet got un-deployed
2021/12/14 19:09:29 No eventing-con process running
2021/12/14 19:09:29 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/14 19:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:09:41 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/14 19:09:46 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/14 19:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:09:58 Waiting for dst bucket item count to get to: 0 curr count: 52
2021/12/14 19:10:03 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/14 19:10:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:10:03 Testing get enoent operation on source bucket
2021/12/14 19:10:03 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:10:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:10:09 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:10:14 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:10:19 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:10:24 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 19:10:24 App: TestEnoentAdvancedGet got deployed
2021/12/14 19:10:24 Starting bucket ops to source bucket
2021/12/14 19:10:29 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 19:10:34 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 19:10:34 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 19:10:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:10:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:10:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:10:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:10:44 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:10:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:10:54 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 19:10:54 App: TestEnoentAdvancedGet got un-deployed
2021/12/14 19:10:54 No eventing-con process running
2021/12/14 19:10:54 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/14 19:10:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:11:06 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 19:11:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.98s)
=== RUN   TestEnoentAdvancedReplace
2021/12/14 19:11:13 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 19:11:19 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:11:24 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:11:29 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:11:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:11:34 App: TestEnoentAdvancedReplace got deployed
2021/12/14 19:11:34 Starting bucket ops to source bucket
2021/12/14 19:11:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:11:44 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:11:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:11:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:11:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:11:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:11:49 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:11:54 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:11:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:12:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:12:04 App: TestEnoentAdvancedReplace got un-deployed
2021/12/14 19:12:04 No eventing-con process running
2021/12/14 19:12:04 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/14 19:12:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:12:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:12:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:12:28 Testing get enoent operation on source bucket
2021/12/14 19:12:28 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:12:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 19:12:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:12:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:12:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:12:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 19:12:49 App: TestEnoentAdvancedReplace got deployed
2021/12/14 19:12:49 Starting bucket ops to source bucket
2021/12/14 19:12:54 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 19:12:59 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 19:12:59 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 19:12:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:12:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:12:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:13:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:13:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:13:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:13:19 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 19:13:19 App: TestEnoentAdvancedReplace got un-deployed
2021/12/14 19:13:19 No eventing-con process running
2021/12/14 19:13:19 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/14 19:13:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:13:31 Waiting for dst bucket item count to get to: 0 curr count: 195
2021/12/14 19:13:36 Waiting for dst bucket item count to get to: 0 curr count: 195
2021/12/14 19:13:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:13:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.96s)
=== RUN   TestEnoentAdvancedDelete
2021/12/14 19:13:43 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:13:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 19:13:49 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:13:54 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:13:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:14:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:14:04 App: TestEnoentAdvancedDelete got deployed
2021/12/14 19:14:04 Starting bucket ops to source bucket
2021/12/14 19:14:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:14:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:14:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:14:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:14:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:14:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:14:19 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:14:24 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:14:29 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:14:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:14:34 App: TestEnoentAdvancedDelete got un-deployed
2021/12/14 19:14:34 No eventing-con process running
2021/12/14 19:14:34 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/14 19:14:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:14:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:14:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:14:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:14:58 Testing get enoent operation on source bucket
2021/12/14 19:14:58 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:14:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 19:15:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:15:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:15:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:15:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 19:15:19 App: TestEnoentAdvancedDelete got deployed
2021/12/14 19:15:19 Starting bucket ops to source bucket
2021/12/14 19:15:24 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 19:15:29 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 19:15:29 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 19:15:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:15:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:15:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:15:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:15:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:15:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:15:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 19:15:49 App: TestEnoentAdvancedDelete got un-deployed
2021/12/14 19:15:49 No eventing-con process running
2021/12/14 19:15:49 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/14 19:15:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:16:01 Waiting for dst bucket item count to get to: 0 curr count: 195
2021/12/14 19:16:06 Waiting for dst bucket item count to get to: 0 curr count: 195
2021/12/14 19:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:16:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.02s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/14 19:16:13 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:16:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/14 19:16:19 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:16:24 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:16:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:16:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:16:34 App: TestAdvancedInsertKeyExist got deployed
2021/12/14 19:16:34 Starting bucket ops to source bucket
2021/12/14 19:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:16:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:16:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:16:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:16:39 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:16:44 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:16:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:16:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:16:54 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/14 19:16:54 No eventing-con process running
2021/12/14 19:16:54 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/14 19:16:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:17:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:17:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:17:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:17:08 Pumping document in the destination bucket
2021/12/14 19:17:08 Starting bucket ops to source bucket
2021/12/14 19:17:08 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:17:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/14 19:17:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:17:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:17:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:17:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 19:17:30 App: TestAdvancedInsertKeyExist got deployed
2021/12/14 19:17:30 Starting bucket ops to source bucket
2021/12/14 19:17:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:17:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:17:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:17:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:17:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:17:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:17:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:17:50 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:17:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:18:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:18:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 19:18:05 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/14 19:18:05 No eventing-con process running
2021/12/14 19:18:05 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/14 19:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:18:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:18:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:18:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.70s)
=== RUN   TestExipryGet
2021/12/14 19:18:23 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:18:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:18:30 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:18:35 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:18:40 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:18:45 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:18:45 App: TestExipryGet got deployed
2021/12/14 19:18:45 Starting bucket ops to source bucket
2021/12/14 19:19:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:19:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:19:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:19:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:19:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:19:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:19:40 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:19:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:19:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:19:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:19:55 App: TestExipryGet got un-deployed
2021/12/14 19:19:55 No eventing-con process running
2021/12/14 19:19:55 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/14 19:19:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:20:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:20:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:20:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:20:09 Pumping document in the destination bucket for expiry get
2021/12/14 19:20:09 Starting bucket ops to source bucket
2021/12/14 19:20:09 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:20:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:20:15 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:20:20 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:20:25 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:20:30 Waiting for app: TestExipryGet to get deployed
2021/12/14 19:20:30 App: TestExipryGet got deployed
2021/12/14 19:20:30 Starting bucket ops to source bucket
2021/12/14 19:21:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:21:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:21:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:21:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:21:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:21:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:21:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:21:31 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:21:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:21:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 19:21:41 App: TestExipryGet got un-deployed
2021/12/14 19:21:41 No eventing-con process running
2021/12/14 19:21:41 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/14 19:21:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:21:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:21:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:21:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:21:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:22:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.84s)
=== RUN   TestExipryInsert
2021/12/14 19:22:04 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:22:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/14 19:22:10 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:22:15 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:22:20 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:22:25 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:22:25 App: TestExipryInsert got deployed
2021/12/14 19:22:25 Starting bucket ops to source bucket
2021/12/14 19:23:11 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 19:23:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:23:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:23:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:23:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:23:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:23:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:23:26 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:23:31 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:23:31 App: TestExipryInsert got un-deployed
2021/12/14 19:23:31 No eventing-con process running
2021/12/14 19:23:31 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/14 19:23:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:23:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:23:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:23:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:23:45 Testing on the destination bucket for expiry insert
2021/12/14 19:23:45 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:23:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/14 19:23:51 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:23:56 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:24:01 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:24:06 Waiting for app: TestExipryInsert to get deployed
2021/12/14 19:24:06 App: TestExipryInsert got deployed
2021/12/14 19:24:06 Starting bucket ops to source bucket
2021/12/14 19:24:51 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:24:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:24:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:24:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:24:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:24:56 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:25:01 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:25:06 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:25:11 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 19:25:11 App: TestExipryInsert got un-deployed
2021/12/14 19:25:11 No eventing-con process running
2021/12/14 19:25:11 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/14 19:25:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:25:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:25:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:25:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:25:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.69s)
=== RUN   TestExipryReplace
2021/12/14 19:25:35 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:25:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 19:25:41 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:25:46 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:25:51 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:25:56 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:25:56 App: TestExipryReplace got deployed
2021/12/14 19:25:56 Starting bucket ops to source bucket
2021/12/14 19:26:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:26:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:26:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:26:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:26:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:26:46 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:26:51 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:26:56 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:27:01 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:27:01 App: TestExipryReplace got un-deployed
2021/12/14 19:27:01 No eventing-con process running
2021/12/14 19:27:01 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/14 19:27:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:27:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:27:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:27:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:27:15 Testing on the destination bucket for expiry insert
2021/12/14 19:27:15 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:27:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 19:27:22 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:27:27 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:27:32 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:27:37 Waiting for app: TestExipryReplace to get deployed
2021/12/14 19:27:37 App: TestExipryReplace got deployed
2021/12/14 19:27:37 Starting bucket ops to source bucket
2021/12/14 19:28:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:28:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:28:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:28:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:28:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:28:27 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:28:32 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:28:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:28:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 19:28:42 App: TestExipryReplace got un-deployed
2021/12/14 19:28:42 No eventing-con process running
2021/12/14 19:28:42 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/14 19:28:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:28:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:28:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:28:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:29:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.56s)
=== RUN   TestExipryUpsert
2021/12/14 19:29:05 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:29:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/14 19:29:12 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:29:17 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:29:22 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:29:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:29:27 App: TestExipryUpsert got deployed
2021/12/14 19:29:27 Starting bucket ops to source bucket
2021/12/14 19:30:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:30:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:30:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:30:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:30:17 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:30:22 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:30:27 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:30:32 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:30:32 App: TestExipryUpsert got un-deployed
2021/12/14 19:30:32 No eventing-con process running
2021/12/14 19:30:32 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/14 19:30:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:30:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:30:46 Testing on the destination bucket for expiry insert
2021/12/14 19:30:46 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:30:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/14 19:30:52 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:30:57 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:31:02 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:31:07 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 19:31:07 App: TestExipryUpsert got deployed
2021/12/14 19:31:07 Starting bucket ops to source bucket
2021/12/14 19:31:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:31:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:32:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:32:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:32:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:32:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:32:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:32:08 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:32:13 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:32:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:32:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 19:32:23 App: TestExipryUpsert got un-deployed
2021/12/14 19:32:23 No eventing-con process running
2021/12/14 19:32:23 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/14 19:32:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:32:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:32:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:32:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:32:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (215.72s)
=== RUN   TestCasReplace
2021/12/14 19:32:41 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:32:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 19:32:47 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:32:52 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:32:57 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:33:02 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:33:02 App: TestCasReplace got deployed
2021/12/14 19:33:02 Starting bucket ops to source bucket
2021/12/14 19:33:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:33:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:33:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:33:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:33:08 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:33:13 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:33:18 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:33:23 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:33:23 App: TestCasReplace got un-deployed
2021/12/14 19:33:23 No eventing-con process running
2021/12/14 19:33:23 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/14 19:33:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:33:35 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/14 19:33:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:33:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:33:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:33:41 Testing on the destination bucket for upsert with cas
2021/12/14 19:33:41 Starting bucket ops to source bucket
2021/12/14 19:33:41 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:33:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 19:33:47 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:33:52 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:33:57 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:34:02 Waiting for app: TestCasReplace to get deployed
2021/12/14 19:34:02 App: TestCasReplace got deployed
2021/12/14 19:34:02 Starting bucket ops to source bucket
2021/12/14 19:34:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:34:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:34:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:34:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:34:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:34:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:34:18 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:34:23 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:34:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:34:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 19:34:33 App: TestCasReplace got un-deployed
2021/12/14 19:34:33 No eventing-con process running
2021/12/14 19:34:33 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/14 19:34:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:34:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:34:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:34:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.19s)
=== RUN   TestCasDelete
2021/12/14 19:34:51 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:34:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 19:34:58 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:35:03 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:35:08 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:35:13 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:35:13 App: TestCasDelete got deployed
2021/12/14 19:35:13 Starting bucket ops to source bucket
2021/12/14 19:35:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:35:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:35:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:35:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:35:18 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:35:23 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:35:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:35:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:35:33 App: TestCasDelete got un-deployed
2021/12/14 19:35:33 No eventing-con process running
2021/12/14 19:35:33 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/14 19:35:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:35:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:35:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:35:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:35:47 Testing on the destination bucket for delete with cas
2021/12/14 19:35:47 Starting bucket ops to source bucket
2021/12/14 19:35:47 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:35:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 19:35:53 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:35:58 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:36:03 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:36:08 Waiting for app: TestCasDelete to get deployed
2021/12/14 19:36:08 App: TestCasDelete got deployed
2021/12/14 19:36:08 Starting bucket ops to source bucket
2021/12/14 19:36:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:36:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:36:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:36:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:36:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 19:36:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:36:24 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:36:29 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:36:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:36:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 19:36:39 App: TestCasDelete got un-deployed
2021/12/14 19:36:39 No eventing-con process running
2021/12/14 19:36:39 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/14 19:36:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:36:51 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/14 19:36:56 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/14 19:36:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:36:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:37:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.94s)
=== RUN   TestCountersIncrement
2021/12/14 19:37:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 19:37:08 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 19:37:08 Removing node(s): [] from the cluster
2021/12/14 19:37:13 Rebalance progress: 100
2021/12/14 19:37:13 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:37:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/14 19:37:20 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:37:25 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:37:30 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:37:35 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:37:35 App: TestCountersIncrement got deployed
2021/12/14 19:37:35 Starting bucket ops to source bucket
2021/12/14 19:37:41 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 19:37:46 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 19:37:46 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 19:37:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:37:51 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:37:56 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:38:01 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:38:06 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:38:06 App: TestCountersIncrement got un-deployed
2021/12/14 19:38:06 No eventing-con process running
2021/12/14 19:38:06 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/14 19:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:38:18 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/14 19:38:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:38:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:38:29 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:38:34 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 19:38:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:38:34 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:38:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/14 19:38:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:38:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:38:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:38:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 19:38:56 App: TestCountersIncrement got deployed
2021/12/14 19:38:56 Starting bucket ops to source bucket
2021/12/14 19:39:02 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 19:39:02 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 19:39:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:39:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:39:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:39:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:39:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:39:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 19:39:22 App: TestCountersIncrement got un-deployed
2021/12/14 19:39:22 No eventing-con process running
2021/12/14 19:39:22 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/14 19:39:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:39:34 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 19:39:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 19:39:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:39:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:39:50 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 19:39:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 19:39:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:39:55 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/14 19:40:01 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.28s)
=== RUN   TestCountersDecrement
2021/12/14 19:40:01 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 19:40:01 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 19:40:01 Removing node(s): [] from the cluster
2021/12/14 19:40:06 Rebalance progress: 100
2021/12/14 19:40:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:40:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/14 19:40:13 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:40:18 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:40:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:40:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:40:28 App: TestCountersDecrement got deployed
2021/12/14 19:40:28 Starting bucket ops to source bucket
2021/12/14 19:40:34 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 19:40:39 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 19:40:44 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 19:40:44 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 19:40:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:40:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:40:49 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:40:54 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:40:59 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:41:04 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:41:04 App: TestCountersDecrement got un-deployed
2021/12/14 19:41:04 No eventing-con process running
2021/12/14 19:41:04 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/14 19:41:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:41:16 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/14 19:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:41:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:41:28 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 19:41:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:41:28 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:41:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/14 19:41:34 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:41:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:41:44 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:41:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 19:41:49 App: TestCountersDecrement got deployed
2021/12/14 19:41:49 Starting bucket ops to source bucket
2021/12/14 19:41:55 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 19:42:00 Waiting for dst bucket item count to get to: 2048 curr count: 261
2021/12/14 19:42:05 Waiting for dst bucket item count to get to: 2048 curr count: 1285
2021/12/14 19:42:05 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 19:42:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:42:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:42:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:42:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:42:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:42:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 19:42:25 App: TestCountersDecrement got un-deployed
2021/12/14 19:42:25 No eventing-con process running
2021/12/14 19:42:25 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/14 19:42:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:42:37 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 19:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:42:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:42:49 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 19:42:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:42:49 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/14 19:42:54 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.32s)
=== RUN   TestCacheGet
2021/12/14 19:42:54 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:42:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:43:00 Waiting for app: TestCacheGet to get deployed
2021/12/14 19:43:05 Waiting for app: TestCacheGet to get deployed
2021/12/14 19:43:10 Waiting for app: TestCacheGet to get deployed
2021/12/14 19:43:15 Waiting for app: TestCacheGet to get deployed
2021/12/14 19:43:15 App: TestCacheGet got deployed
2021/12/14 19:43:15 Starting bucket ops to source bucket
2021/12/14 19:43:20 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 19:43:25 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 19:43:30 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 19:43:35 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/14 19:43:40 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/14 19:43:40 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/14 19:43:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:43:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:43:46 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 19:43:51 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 19:43:56 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 19:44:01 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 19:44:01 App: TestCacheGet got un-deployed
2021/12/14 19:44:01 No eventing-con process running
2021/12/14 19:44:01 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/14 19:44:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:44:12 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/14 19:44:17 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/14 19:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:44:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:44:29 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/14 19:44:34 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/14 19:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.28s)
=== RUN   TestCacheRYOW
2021/12/14 19:44:34 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:44:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/14 19:44:40 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 19:44:45 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 19:44:50 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 19:44:55 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 19:44:55 App: TestCacheRYOW got deployed
2021/12/14 19:44:55 Starting bucket ops to source bucket
2021/12/14 19:45:01 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/14 19:45:01 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/14 19:45:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:45:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:45:06 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 19:45:11 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 19:45:16 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 19:45:21 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 19:45:21 App: TestCacheRYOW got un-deployed
2021/12/14 19:45:21 No eventing-con process running
2021/12/14 19:45:21 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/14 19:45:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:45:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:45:38 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 19:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:45:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:45:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:45:54 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 19:45:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.19s)
=== RUN   TestCacheExpiry
2021/12/14 19:45:54 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:45:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/14 19:46:01 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 19:46:06 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 19:46:11 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 19:46:16 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 19:46:16 App: TestCacheExpiry got deployed
2021/12/14 19:46:16 Starting bucket ops to source bucket
2021/12/14 19:46:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:46:26 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 19:46:31 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 19:46:36 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 19:46:41 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 19:46:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:46:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:46:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:46:46 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 19:46:51 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 19:46:56 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 19:47:01 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 19:47:01 App: TestCacheExpiry got un-deployed
2021/12/14 19:47:01 No eventing-con process running
2021/12/14 19:47:01 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/14 19:47:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:47:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:47:18 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 19:47:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:47:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:47:30 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:47:35 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:47:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.33s)
=== RUN   TestCacheExpiryCustom
2021/12/14 19:47:35 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:47:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/14 19:47:41 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 19:47:46 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 19:47:51 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 19:47:56 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 19:47:56 App: TestCacheExpiryCustom got deployed
2021/12/14 19:47:56 Starting bucket ops to source bucket
2021/12/14 19:48:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:48:06 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 19:48:11 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/14 19:48:16 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 19:48:21 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/14 19:48:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:48:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:48:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:48:26 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 19:48:31 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 19:48:36 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 19:48:41 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 19:48:41 App: TestCacheExpiryCustom got un-deployed
2021/12/14 19:48:41 No eventing-con process running
2021/12/14 19:48:41 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/14 19:48:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:48:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:48:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 19:48:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:49:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:49:10 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/14 19:49:15 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/14 19:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.26s)
=== RUN   TestCacheOverflowCustom
2021/12/14 19:49:15 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:49:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/14 19:49:21 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 19:49:26 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 19:49:31 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 19:49:36 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 19:49:36 App: TestCacheOverflowCustom got deployed
2021/12/14 19:49:36 Starting bucket ops to source bucket
2021/12/14 19:49:41 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/14 19:49:46 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/14 19:49:51 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/14 19:49:56 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/14 19:50:01 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/14 19:50:01 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/14 19:50:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:50:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:50:06 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 19:50:11 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 19:50:17 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 19:50:22 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 19:50:22 App: TestCacheOverflowCustom got un-deployed
2021/12/14 19:50:22 No eventing-con process running
2021/12/14 19:50:22 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/14 19:50:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:50:33 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/14 19:50:38 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/14 19:50:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:50:50 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 19:50:55 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 19:50:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.29s)
=== RUN   TestStrictMode
2021/12/14 19:50:55 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:50:56 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/14 19:50:56 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:50:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/14 19:51:02 Waiting for app: TestDataTypes to get deployed
2021/12/14 19:51:07 Waiting for app: TestDataTypes to get deployed
2021/12/14 19:51:12 Waiting for app: TestDataTypes to get deployed
2021/12/14 19:51:17 Waiting for app: TestDataTypes to get deployed
2021/12/14 19:51:18 App: TestDataTypes got deployed
2021/12/14 19:51:18 Starting bucket ops to source bucket
2021/12/14 19:51:23 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/14 19:51:28 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/14 19:51:33 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/14 19:51:33 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/14 19:51:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:51:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:51:38 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 19:51:43 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 19:51:48 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 19:51:53 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 19:51:53 App: TestDataTypes got un-deployed
2021/12/14 19:51:53 No eventing-con process running
2021/12/14 19:51:53 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/14 19:51:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:52:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 19:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:52:16 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/14 19:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.08s)
=== RUN   TestEnoentGet
2021/12/14 19:52:16 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:52:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:52:22 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:52:28 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:52:33 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:52:38 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:52:38 App: TestEnoentGet got deployed
2021/12/14 19:52:38 Starting bucket ops to source bucket
2021/12/14 19:52:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:52:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:52:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:52:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:52:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:52:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:52:58 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:53:03 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:53:08 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:53:13 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:53:13 App: TestEnoentGet got un-deployed
2021/12/14 19:53:13 No eventing-con process running
2021/12/14 19:53:13 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/14 19:53:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:53:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:53:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:53:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:53:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:53:37 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:53:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/14 19:53:43 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:53:48 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:53:53 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:53:58 Waiting for app: TestEnoentGet to get deployed
2021/12/14 19:53:58 App: TestEnoentGet got deployed
2021/12/14 19:53:58 Starting bucket ops to source bucket
2021/12/14 19:54:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:54:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:54:13 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:54:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:54:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:54:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:54:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:54:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:54:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:54:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 19:54:33 App: TestEnoentGet got un-deployed
2021/12/14 19:54:33 No eventing-con process running
2021/12/14 19:54:33 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/14 19:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:54:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:54:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:54:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:54:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.38s)
=== RUN   TestEnoentDelete
2021/12/14 19:54:57 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:54:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 19:55:03 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:55:08 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:55:13 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:55:18 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:55:18 App: TestEnoentDelete got deployed
2021/12/14 19:55:18 Starting bucket ops to source bucket
2021/12/14 19:55:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:55:28 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:55:33 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:55:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:55:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:55:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:55:38 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:55:43 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:55:48 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:55:53 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:55:53 App: TestEnoentDelete got un-deployed
2021/12/14 19:55:53 No eventing-con process running
2021/12/14 19:55:53 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/14 19:55:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:56:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:56:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:56:17 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:56:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 19:56:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:56:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:56:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:56:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 19:56:38 App: TestEnoentDelete got deployed
2021/12/14 19:56:38 Starting bucket ops to source bucket
2021/12/14 19:56:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 19:56:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 19:56:48 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 19:56:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:56:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:56:53 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:56:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:57:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:57:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 19:57:08 App: TestEnoentDelete got un-deployed
2021/12/14 19:57:08 No eventing-con process running
2021/12/14 19:57:08 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/14 19:57:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:57:20 Waiting for dst bucket item count to get to: 0 curr count: 70
2021/12/14 19:57:25 Waiting for dst bucket item count to get to: 0 curr count: 70
2021/12/14 19:57:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:57:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:57:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:57:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.24s)
=== RUN   TestError
2021/12/14 19:57:47 Waiting for app: TestError to get un-deployed
2021/12/14 19:57:47 App: TestError got un-deployed
2021/12/14 19:57:47 No eventing-con process running
2021/12/14 19:57:47 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/14 19:57:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:57:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:57:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:58:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:58:01 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:58:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/14 19:58:07 Waiting for app: TestError to get deployed
2021/12/14 19:58:12 Waiting for app: TestError to get deployed
2021/12/14 19:58:17 Waiting for app: TestError to get deployed
2021/12/14 19:58:22 Waiting for app: TestError to get deployed
2021/12/14 19:58:22 App: TestError got deployed
2021/12/14 19:58:22 Starting bucket ops to source bucket
2021/12/14 19:58:27 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/14 19:58:32 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/14 19:58:32 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/14 19:58:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 19:58:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 19:58:37 Waiting for app: TestError to get un-deployed
2021/12/14 19:58:42 Waiting for app: TestError to get un-deployed
2021/12/14 19:58:47 Waiting for app: TestError to get un-deployed
2021/12/14 19:58:52 Waiting for app: TestError to get un-deployed
2021/12/14 19:58:52 App: TestError got un-deployed
2021/12/14 19:58:52 No eventing-con process running
2021/12/14 19:58:52 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/14 19:58:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:59:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 19:59:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 19:59:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:59:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:59:21 Waiting for dst bucket item count to get to: 0 curr count: 268
2021/12/14 19:59:26 Waiting for dst bucket item count to get to: 0 curr count: 268
2021/12/14 19:59:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.74s)
=== RUN   TestCRLF
2021/12/14 19:59:36 Waiting for app: TestCRLF to get un-deployed
2021/12/14 19:59:36 App: TestCRLF got un-deployed
2021/12/14 19:59:36 No eventing-con process running
2021/12/14 19:59:36 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/14 19:59:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:59:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:59:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 19:59:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 19:59:49 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/14 19:59:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/14 19:59:56 Waiting for app: TestCRLF to get deployed
2021/12/14 20:00:01 Waiting for app: TestCRLF to get deployed
2021/12/14 20:00:06 Waiting for app: TestCRLF to get deployed
2021/12/14 20:00:11 Waiting for app: TestCRLF to get deployed
2021/12/14 20:00:11 App: TestCRLF got deployed
2021/12/14 20:00:11 Starting bucket ops to source bucket
2021/12/14 20:00:16 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 20:00:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 20:00:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 20:00:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:00:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:00:26 Waiting for app: TestCRLF to get un-deployed
2021/12/14 20:00:31 Waiting for app: TestCRLF to get un-deployed
2021/12/14 20:00:36 Waiting for app: TestCRLF to get un-deployed
2021/12/14 20:00:41 Waiting for app: TestCRLF to get un-deployed
2021/12/14 20:00:41 App: TestCRLF got un-deployed
2021/12/14 20:00:41 No eventing-con process running
2021/12/14 20:00:41 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/14 20:00:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:00:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 20:00:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 20:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:01:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:01:10 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/14 20:01:15 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/14 20:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.84s)
=== RUN   TestImportExport
2021/12/14 20:01:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 20:01:25 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 20:01:25 No eventing-con process running
2021/12/14 20:01:25 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/14 20:01:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:01:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:01:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:01:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:01:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 20:01:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 20:01:43 No eventing-con process running
2021/12/14 20:01:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/14 20:01:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:01:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:01:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:01:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:01:57 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:01:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 20:01:58 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:01:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 20:02:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 20:02:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 20:02:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 20:02:19 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 20:02:19 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/14 20:02:24 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 20:02:29 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 20:02:34 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 20:02:34 App: TestImportExport_bucket_op_on_update got deployed
2021/12/14 20:02:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:02:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 20:02:44 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 20:02:49 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 20:02:54 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 20:02:54 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 20:02:54 No eventing-con process running
2021/12/14 20:02:54 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/14 20:02:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:03:00 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 20:03:05 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 20:03:10 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 20:03:15 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 20:03:15 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 20:03:15 No eventing-con process running
2021/12/14 20:03:15 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/14 20:03:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 20:03:32 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 20:03:32 No eventing-con process running
2021/12/14 20:03:32 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/14 20:03:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:03:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:03:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 20:03:51 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 20:03:51 No eventing-con process running
2021/12/14 20:03:51 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/14 20:03:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:03:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:03:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:04:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.58s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/14 20:04:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:04:14 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 20:04:14 No eventing-con process running
2021/12/14 20:04:14 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/14 20:04:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:04:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:04:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:04:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:04:28 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:04:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:04:29 Starting bucket ops to source bucket
2021/12/14 20:04:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:04:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:04:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:04:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:04:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 20:04:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:05:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 20:05:02 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 20:05:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:05:02 Undeploying app: bucket_op_on_update
2021/12/14 20:05:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:05:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:05:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:05:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:05:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:05:22 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 20:05:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:05:22 No eventing-con process running
2021/12/14 20:05:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:05:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:06:06 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:06:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:06:07 Starting bucket ops to source bucket
2021/12/14 20:06:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:06:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:06:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:06:30 Waiting for dst bucket item count to get to: 5000 curr count: 365
2021/12/14 20:06:35 Waiting for dst bucket item count to get to: 5000 curr count: 365
2021/12/14 20:06:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:06:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 20:06:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 20:06:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:06:40 Undeploying app: bucket_op_on_update
2021/12/14 20:06:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:06:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:06:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:06:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:07:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:07:01 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 20:07:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:07:01 No eventing-con process running
2021/12/14 20:07:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:07:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:07:45 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:07:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:07:46 Starting bucket ops to source bucket
2021/12/14 20:07:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:07:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:08:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:08:09 Waiting for dst bucket item count to get to: 5000 curr count: 2130
2021/12/14 20:08:14 Waiting for dst bucket item count to get to: 5000 curr count: 2130
2021/12/14 20:08:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:08:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 20:08:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 20:08:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:08:19 Undeploying app: bucket_op_on_update
2021/12/14 20:08:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:08:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:08:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:08:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:08:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:08:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 20:08:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:08:39 No eventing-con process running
2021/12/14 20:08:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:08:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:09:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:09:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:09:24 Starting bucket ops to source bucket
2021/12/14 20:09:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:09:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:09:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:09:47 Waiting for dst bucket item count to get to: 5000 curr count: 1352
2021/12/14 20:09:52 Waiting for dst bucket item count to get to: 5000 curr count: 3852
2021/12/14 20:09:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:09:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 20:09:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 20:09:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:09:57 Undeploying app: bucket_op_on_update
2021/12/14 20:09:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:10:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:10:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:10:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:10:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:10:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 20:10:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:10:17 No eventing-con process running
2021/12/14 20:10:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:10:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:11:01 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:11:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:11:03 Starting bucket ops to source bucket
2021/12/14 20:11:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:11:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:11:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:11:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:11:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:11:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 20:11:31 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 20:11:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:11:31 Undeploying app: bucket_op_on_update
2021/12/14 20:11:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:11:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:11:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:11:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:11:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 20:11:51 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 20:11:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:11:51 No eventing-con process running
2021/12/14 20:11:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:12:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:12:35 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.77s)
=== RUN   TestOnUpdateN1QLOp
2021/12/14 20:12:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 20:12:45 App: TestOnUpdateN1QLOp got un-deployed
2021/12/14 20:12:45 No eventing-con process running
2021/12/14 20:12:45 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/14 20:12:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:12:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:12:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:12:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:12:59 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:13:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/14 20:13:00 Starting bucket ops to source bucket
2021/12/14 20:13:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:13:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:13:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:13:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:13:28 Waiting for dst bucket item count to get to: 5000 curr count: 2517
2021/12/14 20:13:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:13:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:13:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:13:33 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 20:13:38 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 20:13:43 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 20:13:48 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 20:13:48 App: TestOnUpdateN1QLOp got un-deployed
2021/12/14 20:13:48 No eventing-con process running
2021/12/14 20:13:48 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/14 20:13:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:14:00 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 20:14:05 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 20:14:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:14:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:14:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:14:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.49s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/14 20:14:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 20:14:26 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/14 20:14:26 No eventing-con process running
2021/12/14 20:14:26 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/14 20:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:14:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:14:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:14:40 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:14:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:14:41 Starting bucket ops to source bucket
2021/12/14 20:14:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:14:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:14:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:15:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:15:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:15:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:15:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:15:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 20:15:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 20:15:19 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 20:15:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 20:15:24 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/14 20:15:24 No eventing-con process running
2021/12/14 20:15:24 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/14 20:15:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:15:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:15:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:15:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:15:48 Waiting for dst bucket item count to get to: 0 curr count: 3619
2021/12/14 20:15:53 Waiting for dst bucket item count to get to: 0 curr count: 1119
2021/12/14 20:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.70s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/14 20:16:03 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 20:16:03 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 20:16:03 No eventing-con process running
2021/12/14 20:16:03 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/14 20:16:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:16:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:16:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:16:17 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:16:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:16:18 Starting bucket ops to source bucket
2021/12/14 20:16:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:16:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:16:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:16:41 Waiting for dst bucket item count to get to: 5000 curr count: 129
2021/12/14 20:16:46 Waiting for dst bucket item count to get to: 5000 curr count: 129
2021/12/14 20:16:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:16:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:16:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:16:51 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 20:16:56 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 20:17:01 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 20:17:06 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 20:17:06 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 20:17:06 No eventing-con process running
2021/12/14 20:17:06 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/14 20:17:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:17:18 Waiting for dst bucket item count to get to: 0 curr count: 3107
2021/12/14 20:17:23 Waiting for dst bucket item count to get to: 0 curr count: 607
2021/12/14 20:17:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:17:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:17:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:17:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.67s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/14 20:17:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 20:17:45 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/14 20:17:45 No eventing-con process running
2021/12/14 20:17:45 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/14 20:17:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:17:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:17:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:17:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:17:58 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:18:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/14 20:18:00 Starting bucket ops to source bucket
2021/12/14 20:18:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:18:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:18:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:18:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:18:28 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 20:18:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:18:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:18:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:18:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 20:18:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 20:18:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 20:18:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 20:18:48 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/14 20:18:48 No eventing-con process running
2021/12/14 20:18:48 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/14 20:18:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:19:00 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 20:19:05 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 20:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:19:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:19:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.70s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/14 20:19:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 20:19:26 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/14 20:19:27 No eventing-con process running
2021/12/14 20:19:27 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/14 20:19:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:19:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:19:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:19:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:19:40 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:19:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/14 20:19:42 Starting bucket ops to source bucket
2021/12/14 20:19:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:19:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:20:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:20:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:20:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:20:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:20:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:20:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 20:20:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 20:20:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 20:20:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 20:20:25 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/14 20:20:25 No eventing-con process running
2021/12/14 20:20:25 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/14 20:20:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:20:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:20:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:20:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:20:49 Waiting for dst bucket item count to get to: 0 curr count: 4454
2021/12/14 20:20:54 Waiting for dst bucket item count to get to: 0 curr count: 1954
2021/12/14 20:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.17s)
=== RUN   TestOnDeleteBucketOp
2021/12/14 20:21:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 20:21:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/14 20:21:04 No eventing-con process running
2021/12/14 20:21:04 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/14 20:21:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:21:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:21:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:21:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:21:17 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:21:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 20:21:18 Starting bucket ops to source bucket
2021/12/14 20:21:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:21:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:21:39 Waiting for dst bucket item count to get to: 5000 curr count: 15
2021/12/14 20:21:44 Waiting for dst bucket item count to get to: 5000 curr count: 2519
2021/12/14 20:21:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:21:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:21:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:21:49 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 20:21:54 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 20:21:59 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 20:22:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 20:22:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/14 20:22:04 No eventing-con process running
2021/12/14 20:22:04 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/14 20:22:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:22:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:22:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:22:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:22:28 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/14 20:22:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.81s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/14 20:22:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 20:22:38 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/14 20:22:38 No eventing-con process running
2021/12/14 20:22:38 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/14 20:22:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:22:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:22:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:22:52 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:22:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/14 20:22:53 Starting bucket ops to source bucket
2021/12/14 20:23:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:23:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:23:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:23:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:23:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:23:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:23:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 20:23:24 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 20:23:29 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 20:23:34 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 20:23:34 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/14 20:23:34 No eventing-con process running
2021/12/14 20:23:34 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/14 20:23:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:23:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:23:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:23:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:23:58 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/14 20:23:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.91s)
=== RUN   TestTimerBucketOp
2021/12/14 20:24:08 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 20:24:08 App: TestTimerBucketOp got un-deployed
2021/12/14 20:24:08 No eventing-con process running
2021/12/14 20:24:08 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/14 20:24:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:24:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:24:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:24:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:24:22 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:24:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 20:24:23 Starting bucket ops to source bucket
2021/12/14 20:24:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:24:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:24:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:24:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:24:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:24:56 Waiting for dst bucket item count to get to: 5000 curr count: 7
2021/12/14 20:25:01 Waiting for dst bucket item count to get to: 5000 curr count: 81
2021/12/14 20:25:06 Waiting for dst bucket item count to get to: 5000 curr count: 2574
2021/12/14 20:25:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:25:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:25:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:25:11 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 20:25:16 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 20:25:21 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 20:25:26 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 20:25:26 App: TestTimerBucketOp got un-deployed
2021/12/14 20:25:26 No eventing-con process running
2021/12/14 20:25:26 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/14 20:25:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:25:38 Waiting for dst bucket item count to get to: 0 curr count: 3529
2021/12/14 20:25:43 Waiting for dst bucket item count to get to: 0 curr count: 1029
2021/12/14 20:25:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:25:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.53s)
=== RUN   TestTimerInPastBucketOp
2021/12/14 20:26:05 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 20:26:05 App: TestTimerInPastBucketOp got un-deployed
2021/12/14 20:26:05 No eventing-con process running
2021/12/14 20:26:05 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/14 20:26:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:26:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:26:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:26:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:26:18 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:26:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 20:26:20 Starting bucket ops to source bucket
2021/12/14 20:26:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:26:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:26:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:26:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:26:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:26:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:26:58 Waiting for dst bucket item count to get to: 5000 curr count: 1195
2021/12/14 20:27:03 Waiting for dst bucket item count to get to: 5000 curr count: 3499
2021/12/14 20:27:08 Waiting for dst bucket item count to get to: 5000 curr count: 4804
2021/12/14 20:27:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:27:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:27:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:27:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 20:27:18 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 20:27:23 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 20:27:28 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 20:27:28 App: TestTimerInPastBucketOp got un-deployed
2021/12/14 20:27:28 No eventing-con process running
2021/12/14 20:27:28 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/14 20:27:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:27:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 20:27:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 20:27:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:27:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:27:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:27:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.31s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/14 20:28:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:28:06 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 20:28:06 No eventing-con process running
2021/12/14 20:28:06 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/14 20:28:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:28:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:28:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:28:20 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:28:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 20:28:21 Starting bucket ops to source bucket
2021/12/14 20:28:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:28:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:28:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:28:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:28:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:28:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:28:59 Waiting for dst bucket item count to get to: 5000 curr count: 2363
2021/12/14 20:29:04 Waiting for dst bucket item count to get to: 5000 curr count: 2363
2021/12/14 20:29:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:29:09 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 20:29:09 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 20:29:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:29:09 Undeploying app: bucket_op_with_timer
2021/12/14 20:29:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:29:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:29:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:29:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:29:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:29:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 20:29:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:29:29 No eventing-con process running
2021/12/14 20:29:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:29:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:30:13 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:30:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 20:30:14 Starting bucket ops to source bucket
2021/12/14 20:30:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:30:58 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 20:30:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:31:03 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 20:31:03 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 20:31:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:31:03 Undeploying app: bucket_op_with_timer
2021/12/14 20:31:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:31:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:31:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:31:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:31:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:31:23 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 20:31:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:31:23 No eventing-con process running
2021/12/14 20:31:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:31:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:32:07 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:32:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 20:32:08 Starting bucket ops to source bucket
2021/12/14 20:32:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:32:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:32:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:32:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:32:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:32:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:32:46 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 20:32:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:32:51 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 20:32:51 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 20:32:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:32:51 Undeploying app: bucket_op_with_timer
2021/12/14 20:32:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:32:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:33:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:33:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:33:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:33:11 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 20:33:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:33:11 No eventing-con process running
2021/12/14 20:33:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:33:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:33:55 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:33:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 20:33:56 Starting bucket ops to source bucket
2021/12/14 20:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:34:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:34:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:34:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:34:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:34:29 Waiting for dst bucket item count to get to: 5000 curr count: 1176
2021/12/14 20:34:34 Waiting for dst bucket item count to get to: 5000 curr count: 1176
2021/12/14 20:34:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:34:39 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 20:34:39 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 20:34:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:34:39 Undeploying app: bucket_op_with_timer
2021/12/14 20:34:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:34:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:34:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:34:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:34:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:34:59 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 20:34:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:34:59 No eventing-con process running
2021/12/14 20:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:35:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:35:44 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:35:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 20:35:45 Starting bucket ops to source bucket
2021/12/14 20:35:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:35:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:36:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:36:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:36:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:36:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:36:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:36:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:36:28 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 20:36:28 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 20:36:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:36:28 Undeploying app: bucket_op_with_timer
2021/12/14 20:36:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:36:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:36:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:36:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:36:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 20:36:48 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 20:36:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:36:48 No eventing-con process running
2021/12/14 20:36:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:36:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:37:32 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (575.80s)
=== RUN   TestMultipleHandlers
2021/12/14 20:37:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 20:37:42 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/14 20:37:42 No eventing-con process running
2021/12/14 20:37:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/14 20:37:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:37:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:37:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:37:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:38:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 20:38:01 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/14 20:38:01 No eventing-con process running
2021/12/14 20:38:01 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/14 20:38:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:38:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:38:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:38:14 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:38:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 20:38:15 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:38:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/14 20:38:21 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 20:38:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 20:38:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 20:38:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 20:38:36 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/14 20:38:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 20:38:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 20:38:51 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 20:38:51 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/14 20:38:51 Starting bucket ops to source bucket
2021/12/14 20:39:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:39:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:39:02 Starting bucket ops to source bucket
2021/12/14 20:39:16 Waiting for dst bucket item count to get to: 10000 curr count: 6627
2021/12/14 20:39:21 Waiting for dst bucket item count to get to: 10000 curr count: 6627
2021/12/14 20:39:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 20:39:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:39:22 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:39:22 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:39:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:39:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 20:39:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 20:39:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 20:39:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 20:39:42 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/14 20:39:42 No eventing-con process running
2021/12/14 20:39:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/14 20:39:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:39:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:39:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:40:01 Waiting for dst bucket item count to get to: 0 curr count: 8914
2021/12/14 20:40:06 Waiting for dst bucket item count to get to: 0 curr count: 8914
2021/12/14 20:40:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:40:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:40:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 20:40:16 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 20:40:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 20:40:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 20:40:26 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/14 20:40:26 No eventing-con process running
2021/12/14 20:40:26 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/14 20:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:40:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:40:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:40:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.81s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/14 20:40:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 20:40:50 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/14 20:40:50 No eventing-con process running
2021/12/14 20:40:50 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/14 20:40:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:40:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:41:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:41:03 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:41:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:41:05 Starting bucket ops to source bucket
2021/12/14 20:41:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:41:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:41:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:41:27 Waiting for dst bucket item count to get to: 5000 curr count: 559
2021/12/14 20:41:32 Waiting for dst bucket item count to get to: 5000 curr count: 3059
2021/12/14 20:41:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:41:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:41:38 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 20:41:38 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 20:41:38 Pausing app: bucket_op_on_update
2021/12/14 20:41:38 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:41:43 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 20:41:43 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 20:41:43 Resuming app: bucket_op_on_update
2021/12/14 20:41:44 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:41:44 Starting bucket ops to source bucket
2021/12/14 20:41:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:41:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:42:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:42:07 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/14 20:42:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 20:42:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:42:12 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 20:42:12 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 20:42:12 Pausing app: bucket_op_on_update
2021/12/14 20:42:12 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:42:17 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 20:42:17 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 20:42:17 Resuming app: bucket_op_on_update
2021/12/14 20:42:18 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:42:18 Starting bucket ops to source bucket
2021/12/14 20:42:26 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 20:42:31 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 20:42:31 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/14 20:42:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:42:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 20:42:36 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 20:42:36 Pausing app: bucket_op_on_update
2021/12/14 20:42:36 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:42:41 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 20:42:41 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 20:42:41 Resuming app: bucket_op_on_update
2021/12/14 20:42:42 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:42:42 Starting bucket ops to source bucket
2021/12/14 20:42:50 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:42:55 Waiting for dst bucket item count to get to: 20000 curr count: 16212
2021/12/14 20:43:00 Waiting for dst bucket item count to get to: 20000 curr count: 16212
2021/12/14 20:43:00 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 20:43:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:43:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 20:43:06 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 20:43:06 Pausing app: bucket_op_on_update
2021/12/14 20:43:06 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:43:11 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 20:43:11 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 20:43:11 Resuming app: bucket_op_on_update
2021/12/14 20:43:12 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:43:12 Starting bucket ops to source bucket
2021/12/14 20:43:20 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:43:25 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:43:30 Waiting for dst bucket item count to get to: 25000 curr count: 21237
2021/12/14 20:43:35 Waiting for dst bucket item count to get to: 25000 curr count: 21237
2021/12/14 20:43:35 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 20:43:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:43:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 20:43:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 20:43:40 Pausing app: bucket_op_on_update
2021/12/14 20:43:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:43:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 20:43:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 20:43:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:43:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 20:43:55 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 20:44:00 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 20:44:05 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 20:44:05 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/14 20:44:05 No eventing-con process running
2021/12/14 20:44:05 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/14 20:44:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:44:17 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 20:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:44:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:44:29 Waiting for dst bucket item count to get to: 0 curr count: 23795
2021/12/14 20:44:34 Waiting for dst bucket item count to get to: 0 curr count: 11295
2021/12/14 20:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.95s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/14 20:44:44 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 20:44:44 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/14 20:44:44 No eventing-con process running
2021/12/14 20:44:44 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/14 20:44:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:44:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:44:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:44:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:44:57 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:44:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 20:44:58 Starting bucket ops to source bucket
2021/12/14 20:45:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:45:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:45:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:45:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:45:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:45:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:45:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 20:45:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:45:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:45:42 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 20:45:42 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 20:45:42 Pausing app: bucket_op_with_timer
2021/12/14 20:45:42 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:45:47 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 20:45:47 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 20:45:47 Resuming app: bucket_op_with_timer
2021/12/14 20:45:48 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:45:48 Starting bucket ops to source bucket
2021/12/14 20:45:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:46:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:46:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:46:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:46:16 Waiting for dst bucket item count to get to: 10000 curr count: 5482
2021/12/14 20:46:21 Waiting for dst bucket item count to get to: 10000 curr count: 6558
2021/12/14 20:46:26 Waiting for dst bucket item count to get to: 10000 curr count: 8576
2021/12/14 20:46:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 20:46:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:46:31 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 20:46:31 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 20:46:31 Pausing app: bucket_op_with_timer
2021/12/14 20:46:31 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:46:36 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 20:46:36 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 20:46:36 Resuming app: bucket_op_with_timer
2021/12/14 20:46:37 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:46:37 Starting bucket ops to source bucket
2021/12/14 20:46:45 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 20:46:50 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 20:46:55 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 20:47:00 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 20:47:05 Waiting for dst bucket item count to get to: 15000 curr count: 10069
2021/12/14 20:47:10 Waiting for dst bucket item count to get to: 15000 curr count: 12304
2021/12/14 20:47:10 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/14 20:47:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:47:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 20:47:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 20:47:15 Pausing app: bucket_op_with_timer
2021/12/14 20:47:15 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:47:20 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 20:47:20 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 20:47:20 Resuming app: bucket_op_with_timer
2021/12/14 20:47:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:47:21 Starting bucket ops to source bucket
2021/12/14 20:47:29 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:47:34 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:47:40 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:47:45 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:47:50 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:47:55 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 20:47:55 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 20:47:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:48:00 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 20:48:00 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 20:48:00 Pausing app: bucket_op_with_timer
2021/12/14 20:48:00 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:48:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 20:48:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 20:48:05 Resuming app: bucket_op_with_timer
2021/12/14 20:48:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:48:06 Starting bucket ops to source bucket
2021/12/14 20:48:14 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:48:19 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:48:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:48:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:48:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 20:48:39 Waiting for dst bucket item count to get to: 25000 curr count: 22698
2021/12/14 20:48:44 Waiting for dst bucket item count to get to: 25000 curr count: 22698
2021/12/14 20:48:44 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 20:48:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:48:49 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 20:48:49 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 20:48:49 Pausing app: bucket_op_with_timer
2021/12/14 20:48:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:48:54 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 20:48:54 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 20:48:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:48:59 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 20:49:04 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 20:49:09 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 20:49:14 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 20:49:14 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/14 20:49:14 No eventing-con process running
2021/12/14 20:49:14 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/14 20:49:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:49:26 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 20:49:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:49:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:49:38 Waiting for dst bucket item count to get to: 0 curr count: 14770
2021/12/14 20:49:43 Waiting for dst bucket item count to get to: 0 curr count: 2270
2021/12/14 20:49:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (308.99s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/14 20:49:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 20:49:53 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/14 20:49:53 No eventing-con process running
2021/12/14 20:49:53 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/14 20:49:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:50:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:50:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:50:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:50:06 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:50:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/14 20:50:12 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 20:50:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 20:50:22 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 20:50:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 20:50:28 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/14 20:50:28 Starting bucket ops to source bucket
2021/12/14 20:50:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:50:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:50:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:50:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:50:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:51:02 Waiting for dst bucket item count to get to: 5000 curr count: 2539
2021/12/14 20:51:07 Waiting for dst bucket item count to get to: 5000 curr count: 2539
2021/12/14 20:51:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:51:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:51:07 Pausing app: bucket_op_with_timer
2021/12/14 20:51:07 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:51:12 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/14 20:51:12 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/14 20:51:12 Resuming app: bucket_op_with_timer
2021/12/14 20:51:13 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:51:13 Starting bucket ops to source bucket
2021/12/14 20:51:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:51:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:51:35 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:51:40 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:51:45 Waiting for dst bucket item count to get to: 10000 curr count: 6668
2021/12/14 20:51:50 Waiting for dst bucket item count to get to: 10000 curr count: 9168
2021/12/14 20:51:50 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 20:51:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:51:55 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 20:52:00 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 20:52:05 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 20:52:10 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 20:52:10 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/14 20:52:10 No eventing-con process running
2021/12/14 20:52:10 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/14 20:52:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:52:22 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 20:52:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 20:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:52:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:52:39 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 20:52:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:52:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.22s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/14 20:52:54 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 20:52:54 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/14 20:52:54 No eventing-con process running
2021/12/14 20:52:54 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/14 20:52:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:53:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:53:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:53:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:53:08 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:53:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 20:53:14 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 20:53:19 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 20:53:24 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 20:53:29 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 20:53:29 App: TestPauseResumeWithEventingReb got deployed
2021/12/14 20:53:29 Starting bucket ops to source bucket
2021/12/14 20:53:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:53:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:53:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:53:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:53:58 Waiting for dst bucket item count to get to: 5000 curr count: 958
2021/12/14 20:54:03 Waiting for dst bucket item count to get to: 5000 curr count: 2175
2021/12/14 20:54:08 Waiting for dst bucket item count to get to: 5000 curr count: 3717
2021/12/14 20:54:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:54:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:54:08 Pausing app: bucket_op_with_timer
2021/12/14 20:54:08 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:54:13 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/14 20:54:13 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/14 20:54:13 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 20:54:14 Removing node(s): [] from the cluster
2021/12/14 20:54:19 Rebalance progress: 29.16666666666666
2021/12/14 20:54:24 Rebalance progress: 29.16666666666666
2021/12/14 20:54:29 Rebalance progress: 100
2021/12/14 20:54:29 Resuming app: bucket_op_with_timer
2021/12/14 20:54:30 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:54:30 Starting bucket ops to source bucket
2021/12/14 20:54:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:54:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:54:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:54:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:55:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 20:55:07 Waiting for dst bucket item count to get to: 10000 curr count: 9985
2021/12/14 20:55:12 Waiting for dst bucket item count to get to: 10000 curr count: 9985
2021/12/14 20:55:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 20:55:12 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 20:55:17 Rebalance progress: 29.16666666666666
2021/12/14 20:55:22 Rebalance progress: 29.16666666666666
2021/12/14 20:55:27 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/14 20:55:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:55:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:56:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:56:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:56:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:56:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:57:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:57:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:57:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 20:57:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:45} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: debccad3054ce1e13b23f65b53ebaf58
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: b4cc40f0ccb047ee87706e2bba28c33f
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
Producer node: 127.0.0.1:9303	Node UUID: 3bf33ce9f4889f32524b4bf659c9a774
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/14 20:58:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:58:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 20:58:07 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 20:58:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 20:58:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 20:58:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 20:58:22 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/14 20:58:22 eventingSetup.go:46: No eventing-con process running
2021/12/14 20:58:22 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/14 20:58:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:58:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 20:58:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 20:58:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:58:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:58:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8979
2021/12/14 20:58:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8979
2021/12/14 20:58:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (371.81s)
=== RUN   TestBucketDeleteAfterPause
2021/12/14 20:59:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/14 20:59:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/14 20:59:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 20:59:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/14 20:59:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:59:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:59:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 20:59:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 20:59:19 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/14 20:59:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/14 20:59:26 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 20:59:31 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 20:59:36 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 20:59:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 20:59:41 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/14 20:59:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 20:59:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 20:59:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 20:59:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 20:59:50 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/14 20:59:50 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 20:59:55 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/14 20:59:56 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/14 20:59:56 handler_test.go:716: Deleting source bucket: default
2021/12/14 21:00:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/14 21:00:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/14 21:00:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/14 21:00:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/14 21:00:07 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:00:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/14 21:00:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:00:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:00:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:00:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:00:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3677
2021/12/14 21:00:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3677
2021/12/14 21:00:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.49s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/14 21:00:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 21:00:45 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/14 21:00:45 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:00:45 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/14 21:00:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:00:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:00:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:00:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:00:58 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:00:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/14 21:01:04 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 21:01:10 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 21:01:15 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 21:01:20 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 21:01:20 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/14 21:01:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:01:54 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/14 21:01:55 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:02:00 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/14 21:02:00 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/14 21:05:00 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/14 21:05:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/14 21:05:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/14 21:05:06 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 21:05:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:05:06 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/14 21:05:11 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 21:05:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:05:11 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2422 expected 5000
2021/12/14 21:05:16 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 21:05:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:05:16 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2802 expected 5000
2021/12/14 21:05:21 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 21:05:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:05:21 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/14 21:05:21 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/14 21:05:21 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:05:31 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 21:05:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 21:05:41 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 21:05:41 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/14 21:05:41 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:05:41 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/14 21:05:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:05:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:05:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:05:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:05:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:06:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.75s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/14 21:06:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 21:06:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/14 21:06:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:06:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/14 21:06:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:06:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:06:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:06:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:06:27 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:06:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 21:06:33 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:06:38 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:06:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:06:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:06:48 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/14 21:06:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:06:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:07:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 585
2021/12/14 21:07:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3085
2021/12/14 21:07:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:07:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:07:07 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/14 21:07:07 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:07:37 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/14 21:07:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 21:07:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:07:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:07:54 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:07:59 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 21:07:59 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/14 21:07:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:08:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:08:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:08:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:08:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:08:16 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/14 21:08:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:08:26 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 21:08:31 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 21:08:37 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 21:08:37 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/14 21:08:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:08:37 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/14 21:08:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:08:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:08:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:08:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.57s)
=== RUN   TestCPPWorkerCleanup
2021/12/14 21:09:00 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 21:09:00 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/14 21:09:00 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:09:00 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/14 21:09:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:09:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:09:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:09:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:09:13 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:09:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/14 21:09:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:09:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:09:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:09:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:09:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 193
2021/12/14 21:09:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1577
2021/12/14 21:09:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3884
2021/12/14 21:09:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:09:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:09:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:09:51 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 21:09:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 21:10:01 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 21:10:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 21:10:06 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/14 21:10:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:10:06 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/14 21:10:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:10:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:10:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:10:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:10:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:10:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:10:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.51s)
=== RUN   TestWithUserXattrs
2021/12/14 21:11:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 21:11:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/14 21:11:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:11:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/14 21:11:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:11:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:11:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:11:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:11:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:11:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:11:31 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:11:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/14 21:11:37 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 21:11:42 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 21:11:47 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 21:11:52 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 21:11:52 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/14 21:11:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:12:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:12:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 21:12:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:12:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:12:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:12:12 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 21:12:17 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 21:12:22 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 21:12:27 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 21:12:27 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/14 21:12:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:12:27 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/14 21:12:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:12:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4073
2021/12/14 21:12:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1573
2021/12/14 21:12:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:12:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:12:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:12:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.76s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/14 21:13:05 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 21:13:05 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 21:13:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:13:05 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/14 21:13:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:13:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:13:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:13:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:13:18 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:13:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/14 21:13:24 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 21:13:29 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 21:13:34 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 21:13:39 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 21:13:39 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/14 21:13:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:13:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:13:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 34
2021/12/14 21:13:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2534
2021/12/14 21:13:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:13:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:14:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:14:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3994
2021/12/14 21:14:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1494
2021/12/14 21:14:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:14:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:14:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:14:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:14:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:14:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:14:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17391
2021/12/14 21:15:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17391
2021/12/14 21:15:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 21:15:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:15:04 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:15:09 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 21:15:14 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 21:15:19 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 21:15:24 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 21:15:24 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 21:15:24 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:15:29 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 21:15:29 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 21:15:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:15:30 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/14 21:15:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:15:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 21:15:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 21:15:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:15:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:15:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 21:15:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.67s)
=== RUN   TestMetaBucketDelete
2021/12/14 21:16:08 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 21:16:08 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 21:16:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:16:08 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/14 21:16:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:16:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:16:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:16:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:16:21 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:16:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 21:16:27 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 21:16:32 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 21:16:37 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 21:16:42 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 21:16:42 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/14 21:16:42 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/14 21:16:43 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/14 21:16:48 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 21:16:48 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 21:17:03 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 21:17:03 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 21:17:03 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:17:03 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/14 21:17:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:17:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:17:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:17:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.61s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/14 21:17:26 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:17:26 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:17:26 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:17:26 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 21:17:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:17:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:17:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:17:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:17:39 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:17:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 21:17:59 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/14 21:18:00 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/14 21:18:15 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:18:15 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:18:30 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:18:30 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:18:30 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:18:30 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 21:18:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:18:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:18:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:18:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.07s)
=== RUN   TestSourceBucketDelete
2021/12/14 21:18:53 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 21:18:53 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 21:18:53 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:18:53 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/14 21:18:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:19:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:19:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:19:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:19:06 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:19:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 21:19:12 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 21:19:17 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 21:19:22 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 21:19:27 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 21:19:27 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/14 21:19:27 handler_test.go:945: Deleting source bucket: default
2021/12/14 21:19:29 handler_test.go:947: Deleted source bucket: default
2021/12/14 21:19:34 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 21:19:34 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 21:19:49 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 21:19:49 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 21:19:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:19:49 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/14 21:19:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:19:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:19:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:20:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.17s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/14 21:20:12 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:20:12 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:20:12 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:20:12 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 21:20:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:20:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:20:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:20:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:20:26 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:20:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 21:20:46 handler_test.go:966: Deleting source bucket: default
2021/12/14 21:20:47 handler_test.go:968: Deleted source bucket: default
2021/12/14 21:21:02 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:21:02 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:21:17 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:21:17 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:21:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:21:17 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 21:21:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:21:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:21:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:21:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.60s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/14 21:21:40 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:21:40 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:21:40 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:21:40 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 21:21:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:21:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:21:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:21:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:21:53 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:21:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 21:22:13 handler_test.go:988: Deleting source bucket: default
2021/12/14 21:22:14 handler_test.go:990: Deleted source bucket: default
2021/12/14 21:22:14 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/14 21:22:15 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/14 21:22:30 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:22:30 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:22:45 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 21:22:45 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 21:22:45 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:22:45 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 21:22:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:22:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:22:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:22:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.34s)
=== RUN   TestUndeployDuringBootstrap
2021/12/14 21:23:08 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 21:23:08 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 21:23:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:23:08 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/14 21:23:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:23:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:23:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:23:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:23:21 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:23:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 21:23:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:23:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:23:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:23:35 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 21:23:35 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/14 21:23:35 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 21:23:40 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/14 21:23:45 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 21:23:50 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 21:23:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 21:23:55 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 21:23:55 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:24:20 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 21:24:20 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 21:24:20 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:24:20 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/14 21:24:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:24:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:24:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:24:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:24:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:24:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2531
2021/12/14 21:24:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 110
2021/12/14 21:24:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (114.65s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/14 21:24:58 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:24:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/14 21:25:04 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 21:25:09 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 21:25:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 21:25:19 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 21:25:19 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/14 21:25:19 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/14 21:25:19 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:25:19 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/14 21:25:24 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/14 21:25:24 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/14 21:25:25 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:25:25 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/14 21:25:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:25:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:25:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:25:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.71s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/14 21:25:48 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 21:25:48 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 21:25:48 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:25:48 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/14 21:25:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:25:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:25:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:26:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:26:01 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:26:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/14 21:26:07 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 21:26:12 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 21:26:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 21:26:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 21:26:22 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/14 21:26:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:26:52 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:26:57 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 21:27:02 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 21:27:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 21:27:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 21:27:13 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 21:27:13 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:28:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 21:28:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 21:28:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:28:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/14 21:28:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:29:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:29:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:29:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:29:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:29:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:29:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.08s)
=== RUN   TestUndeployWithKVFailover
2021/12/14 21:29:36 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 21:29:36 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/14 21:29:36 eventingSetup.go:46: No eventing-con process running
2021/12/14 21:29:36 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/14 21:29:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:29:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:29:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:29:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:29:49 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:29:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/14 21:29:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 21:29:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:30:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:30:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:30:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:30:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:30:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 7
2021/12/14 21:30:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3482
2021/12/14 21:30:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3482
2021/12/14 21:30:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:30:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 21:30:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 21:30:38 Rebalance progress: 29.16666666666666
2021/12/14 21:30:43 Rebalance progress: 29.16666666666666
2021/12/14 21:30:48 Rebalance progress: 100
2021/12/14 21:30:48 Failing over: [127.0.0.1:19003]
2021/12/14 21:30:58 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 21:31:58 Removing node(s): [] from the cluster
2021/12/14 21:32:03 Rebalance progress: 29.16666666666666
2021/12/14 21:32:08 Rebalance progress: 29.16666666666666
2021/12/14 21:32:13 Rebalance progress: 100
2021/12/14 21:32:18 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 21:32:18 App: TestUndeployWithKVFailover got un-deployed
2021/12/14 21:32:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:32:23 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 21:32:23 App: TestUndeployWithKVFailover got un-deployed
2021/12/14 21:32:23 No eventing-con process running
2021/12/14 21:32:23 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/14 21:32:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:32:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:32:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:32:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:32:51 Waiting for dst bucket item count to get to: 0 curr count: 4694
2021/12/14 21:32:56 Waiting for dst bucket item count to get to: 0 curr count: 4694
2021/12/14 21:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.12s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/14 21:33:06 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 21:33:06 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/14 21:33:06 No eventing-con process running
2021/12/14 21:33:06 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/14 21:33:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:33:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:33:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:33:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:33:19 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:33:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/14 21:33:25 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 21:33:30 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 21:33:35 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 21:33:40 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 21:33:40 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/14 21:33:40 Starting bucket ops to source bucket
2021/12/14 21:34:01 Waiting for dst bucket item count to get to: 20000 curr count: 10227
2021/12/14 21:34:01 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 21:34:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:34:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:34:13 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/14 21:34:18 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 21:34:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:34:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:34:30 Waiting for dst bucket item count to get to: 0 curr count: 11924
2021/12/14 21:34:35 Waiting for dst bucket item count to get to: 0 curr count: 11924
2021/12/14 21:34:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:34:35 Starting bucket ops to source bucket
2021/12/14 21:34:48 Waiting for dst bucket item count to get to: 10000 curr count: 4144
2021/12/14 21:34:53 Waiting for dst bucket item count to get to: 10000 curr count: 4144
2021/12/14 21:34:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 21:34:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:34:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:34:58 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 21:35:03 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 21:35:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 21:35:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 21:35:13 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/14 21:35:13 No eventing-con process running
2021/12/14 21:35:13 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/14 21:35:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:35:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 21:35:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:35:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:35:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 21:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.18s)
=== RUN   TestTimerOverwrite
2021/12/14 21:35:46 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 21:35:46 App: TestTimerOverwrite got un-deployed
2021/12/14 21:35:46 No eventing-con process running
2021/12/14 21:35:46 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/14 21:35:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:35:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:35:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:35:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:35:59 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:36:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/14 21:36:05 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 21:36:10 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 21:36:15 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 21:36:20 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 21:36:20 App: TestTimerOverwrite got deployed
2021/12/14 21:36:30 Starting bucket ops to source bucket
2021/12/14 21:36:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:36:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:36:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:36:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 21:37:30 Waiting for dst bucket item count to get to: 5000 curr count: 69
2021/12/14 21:37:35 Waiting for dst bucket item count to get to: 5000 curr count: 69
2021/12/14 21:37:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:37:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:37:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:37:50 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 21:37:55 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 21:38:00 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 21:38:05 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 21:38:05 App: TestTimerOverwrite got un-deployed
2021/12/14 21:38:05 No eventing-con process running
2021/12/14 21:38:05 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/14 21:38:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:38:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:38:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:38:28 Waiting for dst bucket item count to get to: 0 curr count: 3827
2021/12/14 21:38:33 Waiting for dst bucket item count to get to: 0 curr count: 1329
2021/12/14 21:38:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (177.07s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/14 21:38:38 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 21:38:38 Removing node(s): [] from the cluster
2021/12/14 21:38:43 Rebalance progress: 100
2021/12/14 21:38:43 Failing over: [127.0.0.1:19003]
2021/12/14 21:38:49 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 21:38:49 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 21:38:49 No eventing-con process running
2021/12/14 21:38:49 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/14 21:38:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:38:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:39:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:39:02 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:39:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 21:39:03 Starting bucket ops to source bucket
2021/12/14 21:39:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:39:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:39:13 Context: Post to set retry counter, response code: 200 dump: 
2021/12/14 21:39:13 set retry counter, response &{[] }
2021/12/14 21:40:18 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 21:40:18 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 21:40:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:40:18 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/14 21:40:23 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 21:40:23 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 21:40:23 No eventing-con process running
2021/12/14 21:40:23 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/14 21:40:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:40:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:40:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 21:40:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:40:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:40:51 Waiting for dst bucket item count to get to: 0 curr count: 1666
2021/12/14 21:40:56 Waiting for dst bucket item count to get to: 0 curr count: 1666
2021/12/14 21:40:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:40:56 Removing node(s): [] from the cluster
2021/12/14 21:41:01 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (147.96s)
=== RUN   TestOnUpdateSrcMutation
2021/12/14 21:41:11 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 21:41:11 App: TestOnUpdateSrcMutation got un-deployed
2021/12/14 21:41:11 No eventing-con process running
2021/12/14 21:41:11 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/14 21:41:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:41:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:41:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:41:24 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:41:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/14 21:41:25 Starting bucket ops to source bucket
2021/12/14 21:41:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 21:41:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:41:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:41:58 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 21:42:03 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 21:42:08 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 21:42:13 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 21:42:13 App: TestOnUpdateSrcMutation got un-deployed
2021/12/14 21:42:13 No eventing-con process running
2021/12/14 21:42:13 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/14 21:42:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:42:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 21:42:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:42:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:42:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.53s)
=== RUN   TestOnDeleteSrcMutation
2021/12/14 21:42:46 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 21:42:46 App: TestOnDeleteSrcMutation got un-deployed
2021/12/14 21:42:46 No eventing-con process running
2021/12/14 21:42:46 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/14 21:42:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:42:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:42:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:42:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:42:59 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:43:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/14 21:43:00 Starting bucket ops to source bucket
2021/12/14 21:43:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:43:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:43:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:43:34 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 21:43:39 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 21:43:44 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 21:43:49 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 21:43:49 App: TestOnDeleteSrcMutation got un-deployed
2021/12/14 21:43:49 No eventing-con process running
2021/12/14 21:43:49 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/14 21:43:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:44:00 Waiting for dst bucket item count to get to: 0 curr count: 3674
2021/12/14 21:44:05 Waiting for dst bucket item count to get to: 0 curr count: 3674
2021/12/14 21:44:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:44:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:44:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.44s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/14 21:44:22 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 21:44:22 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/14 21:44:22 No eventing-con process running
2021/12/14 21:44:22 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/14 21:44:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:44:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:44:35 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:44:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 21:44:36 Starting bucket ops to source bucket
2021/12/14 21:45:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 21:45:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:45:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:45:18 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 21:45:23 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 21:45:28 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 21:45:33 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 21:45:33 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/14 21:45:33 No eventing-con process running
2021/12/14 21:45:33 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/14 21:45:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:45:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 21:45:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:45:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:45:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.09s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/14 21:46:01 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 21:46:01 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/14 21:46:01 No eventing-con process running
2021/12/14 21:46:01 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/14 21:46:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:46:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:46:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:46:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:46:14 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:46:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 21:46:16 Starting bucket ops to source bucket
2021/12/14 21:46:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 21:46:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 21:46:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:46:59 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 21:47:04 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 21:47:09 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 21:47:14 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 21:47:14 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/14 21:47:14 No eventing-con process running
2021/12/14 21:47:14 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/14 21:47:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:47:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 21:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:47:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:47:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.34s)
=== RUN   TestInterHandlerRecursion
2021/12/14 21:47:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:47:43 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 21:47:43 No eventing-con process running
2021/12/14 21:47:43 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/14 21:47:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:47:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:47:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:48:01 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:48:01 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 21:48:01 No eventing-con process running
2021/12/14 21:48:01 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 21:48:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:48:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:48:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:48:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:48:14 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:48:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/14 21:48:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/14 21:48:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:48:25 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:48:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:48:35 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:48:35 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/14 21:48:35 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:48:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/14 21:48:35 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/14 21:48:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:48:40 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:48:45 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:48:50 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:48:55 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:48:55 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 21:48:55 No eventing-con process running
2021/12/14 21:48:55 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/14 21:48:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:49:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:49:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:49:13 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:49:13 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 21:49:13 No eventing-con process running
2021/12/14 21:49:13 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 21:49:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:49:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:49:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:49:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.68s)
=== RUN   TestInterBucketRecursion
2021/12/14 21:49:31 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:49:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 21:49:32 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 21:49:32 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:49:32 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/14 21:49:37 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 21:49:42 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 21:49:47 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 21:49:52 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 21:49:52 App: TestInterBucketRecursion_function1 got deployed
2021/12/14 21:49:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:49:58 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:50:03 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:50:08 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:50:13 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:50:13 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/14 21:50:13 No eventing-con process running
2021/12/14 21:50:13 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/14 21:50:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:50:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:50:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:50:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:50:31 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/14 21:50:31 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/14 21:50:31 No eventing-con process running
2021/12/14 21:50:31 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/14 21:50:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:50:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:50:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:50:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.57s)
=== RUN   TestLargeHandler
2021/12/14 21:50:44 Changed force_compress value to false
2021/12/14 21:50:44 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:50:44 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/14 21:50:44 Changed force_compress value to true
2021/12/14 21:50:44 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:50:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/14 21:50:51 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 21:50:56 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 21:51:01 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 21:51:06 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 21:51:06 App: TestLargeHandler_function got deployed
2021/12/14 21:51:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:51:11 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 21:51:16 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 21:51:21 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 21:51:21 App: TestLargeHandler_function got un-deployed
2021/12/14 21:51:21 No eventing-con process running
2021/12/14 21:51:21 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/14 21:51:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:51:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:51:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (50.43s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/14 21:51:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:51:44 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 21:51:44 No eventing-con process running
2021/12/14 21:51:44 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/14 21:51:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:51:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:51:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:51:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:52:02 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:52:02 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 21:52:02 No eventing-con process running
2021/12/14 21:52:02 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 21:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:52:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:52:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:52:15 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:52:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/14 21:52:17 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/14 21:52:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:52:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:52:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:52:37 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 21:52:37 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/14 21:52:37 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:52:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/14 21:52:38 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/14 21:52:43 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 21:52:48 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 21:52:53 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 21:52:58 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 21:52:58 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/14 21:52:58 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/14 21:52:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:53:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:53:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:53:13 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:53:18 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 21:53:18 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 21:53:18 No eventing-con process running
2021/12/14 21:53:18 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/14 21:53:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:53:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:53:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:53:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:53:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:53:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:53:42 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:53:47 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:53:52 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 21:53:52 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 21:53:52 No eventing-con process running
2021/12/14 21:53:52 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/14 21:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:54:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:54:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.28s)
=== RUN   TestAllowInterBucketRecursion
2021/12/14 21:54:10 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:54:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 21:54:12 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 21:54:17 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 21:54:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 21:54:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 21:54:32 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 21:54:32 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/14 21:54:32 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:54:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 21:54:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/14 21:54:38 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 21:54:43 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 21:54:48 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 21:54:53 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 21:54:53 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/14 21:54:53 TestAllowInterBucketRecursion_function2 is deployed
2021/12/14 21:54:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:54:58 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:55:03 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:55:08 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:55:13 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 21:55:13 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/14 21:55:13 No eventing-con process running
2021/12/14 21:55:13 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/14 21:55:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:55:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:55:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:55:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:55:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:55:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 21:55:37 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 21:55:42 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 21:55:47 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 21:55:47 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/14 21:55:47 No eventing-con process running
2021/12/14 21:55:47 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/14 21:55:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:55:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:55:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:56:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.92s)
=== RUN   TestInterBucketRecursion2
2021/12/14 21:56:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:56:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:56:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:56:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:56:29 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:56:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 21:56:35 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 21:56:40 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 21:56:45 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 21:56:50 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 21:56:55 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 21:57:00 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 21:57:00 App: TestInterBucketRecursion2_function1 got deployed
2021/12/14 21:57:00 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:57:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 21:57:06 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 21:57:11 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 21:57:16 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 21:57:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 21:57:21 App: TestInterBucketRecursion2_function2 got deployed
2021/12/14 21:57:21 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:57:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 21:57:28 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 21:57:33 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 21:57:38 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 21:57:43 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 21:57:43 App: TestInterBucketRecursion2_function3 got deployed
2021/12/14 21:57:43 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/14 21:57:43 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/14 21:57:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:57:48 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 21:57:53 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 21:57:58 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 21:58:03 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 21:58:03 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/14 21:58:03 No eventing-con process running
2021/12/14 21:58:03 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/14 21:58:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:58:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:58:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:58:21 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 21:58:26 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 21:58:31 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 21:58:36 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 21:58:36 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/14 21:58:36 No eventing-con process running
2021/12/14 21:58:36 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/14 21:58:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:58:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:58:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:58:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:58:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 21:58:55 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 21:59:00 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 21:59:05 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 21:59:10 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 21:59:10 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/14 21:59:10 No eventing-con process running
2021/12/14 21:59:10 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/14 21:59:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:59:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:59:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:59:28 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/14 21:59:28 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/14 21:59:28 No eventing-con process running
2021/12/14 21:59:28 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/14 21:59:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:59:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 21:59:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 21:59:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (222.67s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/14 21:59:48 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 21:59:53 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/14 21:59:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:00:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:00:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:00:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:00:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:01:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:01:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:01:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 22:01:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:55} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/14 22:02:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:02:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 22:02:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 22:02:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/14 22:02:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:02:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 22:02:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 22:02:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/14 22:02:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:02:27 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/14 22:02:27 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/14 22:02:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 22:02:27 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/14 22:02:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:02:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:02:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:02:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:02:45 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:02:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 22:02:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 22:02:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 22:03:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 22:03:06 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 22:03:06 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: b4cc40f0ccb047ee87706e2bba28c33f
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 3bf33ce9f4889f32524b4bf659c9a774
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: debccad3054ce1e13b23f65b53ebaf58
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/14 22:03:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:03:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 22:03:11 Rebalance progress: 29.16666666666666
2021/12/14 22:03:16 Rebalance progress: 29.16666666666666
2021/12/14 22:03:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: debccad3054ce1e13b23f65b53ebaf58
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: b4cc40f0ccb047ee87706e2bba28c33f
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 3bf33ce9f4889f32524b4bf659c9a774
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/14 22:03:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:03:21 handler_test.go:1674: Deleting source bucket: default
2021/12/14 22:03:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 22:03:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 22:03:29 Rebalance progress: 29.16666666666666
2021/12/14 22:03:34 Rebalance progress: 29.16666666666666
2021/12/14 22:03:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 902
	vb list dump: [0-5, 8-10, 12-17, 19-20, 23-25, 27-34, 36-39, 41-42, 44-48, 50-59, 61-101, 103-135, 137, 139-149, 152-157, 159, 161-169, 171, 173-175, 177-183, 185-214, 216-220, 222-224, 226, 228-245, 247-265, 267-277, 279-290, 292-302, 304, 306-311, 313, 316, 318-328, 330, 332-334, 336-356, 358-361, 363, 365-367, 369-390, 392-394, 396-398, 400-403, 405-406, 408-432, 434-443, 445, 447-450, 452-453, 455-457, 460-463, 465-468, 471-477, 481-489, 492, 494-498, 500-505, 507, 509-577, 580-583, 585-587, 589, 591, 593-602, 605-619, 621-626, 629-632, 634, 636, 638-642, 644-687, 689-706, 709, 711-730, 732-742, 744-751, 753-766, 768-770, 772, 774-777, 779-785, 787-793, 795-806, 808-810, 812-815, 817-821, 825, 827-835, 837-857, 859, 861-863, 865-872, 874-886, 888-910, 912, 914-923, 925-936, 939-945, 947, 949, 951-958, 960-983, 985-1010, 1012-1018, 1021-1023]
	stream status: stopped
	len: 122
	vb list dump: [6-7, 11, 18, 21-22, 26, 35, 40, 43, 49, 60, 102, 136, 138, 150-151, 158, 160, 170, 172, 176, 184, 215, 221, 225, 227, 246, 266, 278, 291, 303, 305, 312, 314-315, 317, 329, 331, 335, 357, 362, 364, 368, 391, 395, 399, 404, 407, 433, 444, 446, 451, 454, 458-459, 464, 469-470, 478-480, 490-491, 493, 499, 506, 508, 578-579, 584, 588, 590, 592, 603-604, 620, 627-628, 633, 635, 637, 643, 688, 707-708, 710, 731, 743, 752, 767, 771, 773, 778, 786, 794, 807, 811, 816, 822-824, 826, 836, 858, 860, 864, 873, 887, 911, 913, 924, 937-938, 946, 948, 950, 959, 984, 1011, 1019-1020]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: debccad3054ce1e13b23f65b53ebaf58
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-5, 8-10, 12-17, 19-20, 23-25, 27-34, 36-39, 41-42, 44-48, 50-59, 61-101, 103-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [114-135, 137, 139-149, 152-157, 159, 161-169, 171, 173-175, 177-183, 185-214, 216-220, 222-224, 226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-245, 247-265, 267-277, 279-290, 292-302, 304, 306-311, 313, 316, 318-328, 330, 332-334, 336-341]
Producer node: 172.31.2.134:9300	Node UUID: b4cc40f0ccb047ee87706e2bba28c33f
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [912, 914-923, 925-936, 939-945, 947, 949, 951-958, 960-983, 985-1010, 1012-1018, 1021-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-687, 689-706, 709, 711-730, 732-742, 744-751, 753-766, 768-770, 772, 774-777, 779-785, 787-793, 795-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-806, 808-810, 812-815, 817-821, 825, 827-835, 837-857, 859, 861-863, 865-872, 874-886, 888-910]
Producer node: 127.0.0.1:9303	Node UUID: 3bf33ce9f4889f32524b4bf659c9a774
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-356, 358-361, 363, 365-367, 369-390, 392-394, 396-398, 400-403, 405-406, 408-432, 434-443, 445, 447-450, 452-453, 455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [456-457, 460-463, 465-468, 471-477, 481-489, 492, 494-498, 500-505, 507, 509-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 98
	v: [570-577, 580-583, 585-587, 589, 591, 593-602, 605-619, 621-626, 629-632, 634, 636, 638-642, 644-682]


2021/12/14 22:03:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:03:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/14 22:03:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/14 22:03:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/14 22:03:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/14 22:03:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 22:03:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/14 22:03:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:03:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:03:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:04:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:04:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 22:04:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 22:04:07 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (267.19s)
=== RUN   TestN1QLRecursion
2021/12/14 22:04:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 22:04:11 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 22:04:11 Removing node(s): [] from the cluster
2021/12/14 22:04:16 Rebalance progress: 100
2021/12/14 22:04:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:04:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:04:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:04:39 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:04:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 22:04:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 22:04:46 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 22:04:51 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 22:04:56 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 22:05:01 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 22:05:06 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 22:05:11 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 22:05:11 App: TestN1QLRecursion_function1 got deployed
2021/12/14 22:05:11 TestN1QLRecursion_function1 is deployed
2021/12/14 22:05:11 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:05:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 22:05:12 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/14 22:05:17 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 22:05:22 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 22:05:27 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 22:05:32 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 22:05:32 App: TestN1QLRecursion_function3 got deployed
2021/12/14 22:05:32 TestN1QLRecursion_function3 is deployed
2021/12/14 22:05:32 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:05:32 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/14 22:05:32 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/14 22:05:32 Success: TestN1QLRecursion_function2 is not deployed
2021/12/14 22:05:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:05:37 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 22:05:42 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 22:05:47 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 22:05:52 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 22:05:52 App: TestN1QLRecursion_function1 got un-deployed
2021/12/14 22:05:52 No eventing-con process running
2021/12/14 22:05:52 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/14 22:05:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:05:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:06:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:06:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:06:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:06:11 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 22:06:16 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 22:06:21 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 22:06:26 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 22:06:26 App: TestN1QLRecursion_function3 got un-deployed
2021/12/14 22:06:26 No eventing-con process running
2021/12/14 22:06:26 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/14 22:06:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:06:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:06:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:06:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:06:41 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 22:06:46 Rebalance progress: 100
--- PASS: TestN1QLRecursion (156.04s)
=== RUN   TestN1QLAllowRecursion
2021/12/14 22:06:46 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 22:06:47 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 22:06:47 Removing node(s): [] from the cluster
2021/12/14 22:06:52 Rebalance progress: 100
2021/12/14 22:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:07:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:07:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:07:14 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:07:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 22:07:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 22:07:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 22:07:26 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 22:07:31 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 22:07:36 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 22:07:36 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/14 22:07:36 TestN1QLAllowRecursion_function1 is deployed
2021/12/14 22:07:36 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:07:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 22:07:37 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/14 22:07:42 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 22:07:47 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 22:07:52 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 22:07:57 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 22:07:57 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/14 22:07:57 TestN1QLAllowRecursion_function3 is deployed
2021/12/14 22:07:57 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:07:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 22:07:58 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/14 22:08:03 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 22:08:08 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 22:08:13 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 22:08:18 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 22:08:18 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/14 22:08:18 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/14 22:08:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:08:23 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 22:08:28 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 22:08:33 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 22:08:38 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 22:08:38 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/14 22:08:38 No eventing-con process running
2021/12/14 22:08:38 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/14 22:08:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:08:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:08:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:08:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:08:57 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 22:09:02 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 22:09:07 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 22:09:12 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 22:09:12 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/14 22:09:12 No eventing-con process running
2021/12/14 22:09:12 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/14 22:09:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:09:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:09:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:09:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:09:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:09:31 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 22:09:36 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 22:09:41 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 22:09:46 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 22:09:46 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/14 22:09:46 No eventing-con process running
2021/12/14 22:09:46 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/14 22:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:09:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:09:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:10:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:10:01 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 22:10:06 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.00s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/14 22:10:06 Sleeping for some time
2021/12/14 22:10:16 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 22:10:16 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/14 22:10:16 No eventing-con process running
2021/12/14 22:10:16 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/14 22:10:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:10:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:10:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:10:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:10:30 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:10:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 22:10:31 Starting bucket ops to source bucket
2021/12/14 22:11:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 22:11:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 22:11:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 22:11:25 Starting bucket ops to source bucket
2021/12/14 22:11:26 Starting bucket ops to source bucket
2021/12/14 22:12:32 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/14 22:12:37 Waiting for dst bucket item count to get to: 7000 curr count: 6962
2021/12/14 22:12:42 Waiting for dst bucket item count to get to: 7000 curr count: 6962
2021/12/14 22:12:42 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/14 22:12:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:12:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:12:47 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 22:12:52 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 22:12:57 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 22:13:02 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 22:13:02 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/14 22:13:02 No eventing-con process running
2021/12/14 22:13:02 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/14 22:13:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:13:14 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:13:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:13:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:13:26 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/14 22:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.83s)
=== RUN   TestTimerOverWriteSameReference
2021/12/14 22:13:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 22:13:31 App: TestTimerOverWriteSameReference got un-deployed
2021/12/14 22:13:31 No eventing-con process running
2021/12/14 22:13:31 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/14 22:13:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:13:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:13:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:13:45 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:13:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/14 22:13:51 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 22:13:56 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 22:14:01 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 22:14:06 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 22:14:06 App: TestTimerOverWriteSameReference got deployed
2021/12/14 22:14:16 Starting bucket ops to source bucket
2021/12/14 22:15:17 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/14 22:15:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:15:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:15:32 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 22:15:37 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 22:15:42 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 22:15:47 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 22:15:47 App: TestTimerOverWriteSameReference got un-deployed
2021/12/14 22:15:47 No eventing-con process running
2021/12/14 22:15:47 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/14 22:15:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:15:59 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:16:04 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:16:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:16:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:16:16 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 22:16:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.77s)
=== RUN   TestCancelTimerBucketop
2021/12/14 22:16:21 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 22:16:21 App: TestCancelTimerBucketop got un-deployed
2021/12/14 22:16:21 No eventing-con process running
2021/12/14 22:16:21 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/14 22:16:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:16:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:16:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:16:35 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:16:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/14 22:16:41 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 22:16:46 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 22:16:51 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 22:16:56 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 22:16:56 App: TestCancelTimerBucketop got deployed
2021/12/14 22:16:56 Starting bucket ops to source bucket
2021/12/14 22:17:03 Starting bucket ops to source bucket
2021/12/14 22:18:05 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 22:18:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:18:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:18:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 22:18:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 22:18:20 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 22:18:25 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 22:18:25 App: TestCancelTimerBucketop got un-deployed
2021/12/14 22:18:25 No eventing-con process running
2021/12/14 22:18:26 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/14 22:18:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:18:37 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:18:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:18:49 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:18:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:18:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.39s)
=== RUN   TestJSExpiryDate
2021/12/14 22:18:59 Starting bucket ops to source bucket
2021/12/14 22:19:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 22:19:12 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/14 22:19:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/14 22:19:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:19:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:19:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:19:49 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 22:19:54 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 22:19:59 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 22:20:04 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 22:20:04 App: TestJSExpiryDate got un-deployed
2021/12/14 22:20:04 No eventing-con process running
2021/12/14 22:20:04 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/14 22:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:20:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:20:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.70s)
=== RUN   TestBinaryDoc
2021/12/14 22:20:23 Starting bucket ops to source bucket
2021/12/14 22:20:23 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:20:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/14 22:20:24 Starting bucket ops to source bucket
2021/12/14 22:20:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:20:24 Deleting source bucket docs now
2021/12/14 22:20:24 Starting bucket ops to source bucket
2021/12/14 22:21:01 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/14 22:21:06 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/14 22:21:06 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/14 22:21:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:21:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:21:12 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 22:21:17 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 22:21:22 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 22:21:27 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 22:21:27 App: TestBinaryDoc got un-deployed
2021/12/14 22:21:27 No eventing-con process running
2021/12/14 22:21:27 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/14 22:21:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:21:38 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/14 22:21:43 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/14 22:21:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:21:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:21:55 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/14 22:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.20s)
=== RUN   TestConstantBindings
2021/12/14 22:21:55 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:21:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/14 22:21:56 Starting bucket ops to source bucket
2021/12/14 22:22:02 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 22:22:07 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 22:22:12 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 22:22:17 Waiting for dst bucket item count to get to: 6000 curr count: 1693
2021/12/14 22:22:22 Waiting for dst bucket item count to get to: 6000 curr count: 1693
2021/12/14 22:22:22 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/14 22:22:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:22:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:22:27 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 22:22:32 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 22:22:37 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 22:22:42 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 22:22:42 App: TestConstantBindings got un-deployed
2021/12/14 22:22:42 No eventing-con process running
2021/12/14 22:22:42 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/14 22:22:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:22:54 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:22:59 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:22:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:23:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:23:11 Waiting for dst bucket item count to get to: 0 curr count: 5515
2021/12/14 22:23:16 Waiting for dst bucket item count to get to: 0 curr count: 5515
2021/12/14 22:23:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.68s)
=== RUN   TestCodeInsight
2021/12/14 22:23:21 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 22:23:21 App: TestCodeInsight got un-deployed
2021/12/14 22:23:21 No eventing-con process running
2021/12/14 22:23:21 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/14 22:23:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:23:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:23:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:23:34 Starting bucket ops to source bucket
2021/12/14 22:23:35 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:23:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/14 22:23:41 Waiting for app: TestCodeInsight to get deployed
2021/12/14 22:23:46 Waiting for app: TestCodeInsight to get deployed
2021/12/14 22:23:51 Waiting for app: TestCodeInsight to get deployed
2021/12/14 22:23:56 Waiting for app: TestCodeInsight to get deployed
2021/12/14 22:23:56 App: TestCodeInsight got deployed
2021/12/14 22:24:01 Waiting for dst bucket item count to get to: 1000 curr count: 435
2021/12/14 22:24:01 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 22:24:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:24:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 22:24:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 22:24:16 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 22:24:21 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 22:24:21 App: TestCodeInsight got un-deployed
2021/12/14 22:24:21 No eventing-con process running
2021/12/14 22:24:22 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/14 22:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:24:33 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:24:38 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:24:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:24:50 Waiting for dst bucket item count to get to: 0 curr count: 698
2021/12/14 22:24:55 Waiting for dst bucket item count to get to: 0 curr count: 698
2021/12/14 22:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.28s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12634.332s

Functional tests took 3 hours, 30 minutes and 38 seconds.

N1ql tests

2021/12/14 22:25:03 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/14 22:25:14 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 22:25:14 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 22:25:14 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 22:25:14 Removing node(s): [] from the cluster
2021/12/14 22:25:19 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/14 22:25:24 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/14 22:25:24 App: TestRecursiveMutationN1QL got un-deployed
2021/12/14 22:25:24 No eventing-con process running
2021/12/14 22:25:24 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/14 22:25:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:25:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:25:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:25:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:25:38 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:25:38 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.55s)
=== RUN   TestFlexReset1
2021/12/14 22:25:43 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 22:25:43 App: TestFlexReset1 got un-deployed
2021/12/14 22:25:43 No eventing-con process running
2021/12/14 22:25:43 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/14 22:25:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:25:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:25:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:25:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:25:56 Starting bucket ops to source bucket
2021/12/14 22:25:57 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:25:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/14 22:26:03 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 22:26:08 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 22:26:13 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 22:26:18 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 22:26:18 App: TestFlexReset1 got deployed
2021/12/14 22:26:23 Waiting for dst bucket item count to get to: 1000 curr count: 2
2021/12/14 22:26:23 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 22:26:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:26:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:26:28 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 22:26:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 22:26:38 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 22:26:43 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 22:26:43 App: TestFlexReset1 got un-deployed
2021/12/14 22:26:43 No eventing-con process running
2021/12/14 22:26:43 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/14 22:26:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:26:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:26:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:27:07 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.35s)
=== RUN   TestFlexReset2
2021/12/14 22:27:12 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 22:27:12 App: TestFlexReset2 got un-deployed
2021/12/14 22:27:12 No eventing-con process running
2021/12/14 22:27:12 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/14 22:27:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:27:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:27:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:27:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:27:26 Starting bucket ops to source bucket
2021/12/14 22:27:26 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:27:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/14 22:27:32 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 22:27:37 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 22:27:42 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 22:27:47 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 22:27:47 App: TestFlexReset2 got deployed
2021/12/14 22:27:52 Waiting for dst bucket item count to get to: 1000 curr count: 6
2021/12/14 22:27:52 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 22:27:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:27:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:27:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 22:28:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 22:28:08 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 22:28:13 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 22:28:13 App: TestFlexReset2 got un-deployed
2021/12/14 22:28:13 No eventing-con process running
2021/12/14 22:28:13 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/14 22:28:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:28:24 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:28:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:28:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:28:36 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 22:28:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.29s)
=== RUN   TestN1QLLabelledBreak
2021/12/14 22:28:41 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 22:28:41 App: TestN1QLLabelledBreak got un-deployed
2021/12/14 22:28:41 No eventing-con process running
2021/12/14 22:28:41 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/14 22:28:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:28:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:28:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:28:55 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:28:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/14 22:29:01 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 22:29:06 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 22:29:11 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 22:29:16 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 22:29:16 App: TestN1QLLabelledBreak got deployed
2021/12/14 22:29:16 Starting bucket ops to source bucket
2021/12/14 22:29:26 Waiting for dst bucket item count to get to: 10000 curr count: 285
2021/12/14 22:29:31 Waiting for dst bucket item count to get to: 10000 curr count: 285
2021/12/14 22:29:36 Waiting for dst bucket item count to get to: 10000 curr count: 285
2021/12/14 22:29:41 Waiting for dst bucket item count to get to: 10000 curr count: 3665
2021/12/14 22:29:46 Waiting for dst bucket item count to get to: 10000 curr count: 3665
2021/12/14 22:29:51 Waiting for dst bucket item count to get to: 10000 curr count: 7627
2021/12/14 22:29:56 Waiting for dst bucket item count to get to: 10000 curr count: 7627
2021/12/14 22:29:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 22:29:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:29:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:30:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 22:30:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 22:30:11 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 22:30:16 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 22:30:16 App: TestN1QLLabelledBreak got un-deployed
2021/12/14 22:30:16 No eventing-con process running
2021/12/14 22:30:16 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/14 22:30:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:30:28 Waiting for dst bucket item count to get to: 0 curr count: 3011
2021/12/14 22:30:33 Waiting for dst bucket item count to get to: 0 curr count: 511
2021/12/14 22:30:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:30:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:30:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 22:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.35s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/14 22:30:50 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 22:30:50 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/14 22:30:50 No eventing-con process running
2021/12/14 22:30:50 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/14 22:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:30:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:31:03 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:31:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/14 22:31:09 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 22:31:14 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 22:31:19 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 22:31:24 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 22:31:24 App: TestN1QLUnlabelledBreak got deployed
2021/12/14 22:31:24 Starting bucket ops to source bucket
2021/12/14 22:31:34 Waiting for dst bucket item count to get to: 10000 curr count: 304
2021/12/14 22:31:39 Waiting for dst bucket item count to get to: 10000 curr count: 304
2021/12/14 22:31:44 Waiting for dst bucket item count to get to: 10000 curr count: 882
2021/12/14 22:31:49 Waiting for dst bucket item count to get to: 10000 curr count: 3436
2021/12/14 22:31:54 Waiting for dst bucket item count to get to: 10000 curr count: 6340
2021/12/14 22:31:59 Waiting for dst bucket item count to get to: 10000 curr count: 8482
2021/12/14 22:31:59 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 22:31:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:31:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:32:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 22:32:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 22:32:14 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 22:32:19 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 22:32:19 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/14 22:32:19 No eventing-con process running
2021/12/14 22:32:19 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/14 22:32:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:32:31 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/14 22:32:36 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/14 22:32:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:32:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:32:48 Waiting for dst bucket item count to get to: 0 curr count: 6500
2021/12/14 22:32:53 Waiting for dst bucket item count to get to: 0 curr count: 1500
2021/12/14 22:32:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.23s)
=== RUN   TestN1QLThrowStatement
2021/12/14 22:32:58 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 22:32:58 App: TestN1QLThrowStatement got un-deployed
2021/12/14 22:32:58 No eventing-con process running
2021/12/14 22:32:58 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/14 22:33:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:33:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:33:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:33:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:33:11 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:33:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/14 22:33:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 22:33:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 22:33:28 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 22:33:33 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 22:33:33 App: TestN1QLThrowStatement got deployed
2021/12/14 22:33:33 Starting bucket ops to source bucket
2021/12/14 22:33:42 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 22:33:47 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 22:33:52 Waiting for dst bucket item count to get to: 10000 curr count: 1063
2021/12/14 22:33:57 Waiting for dst bucket item count to get to: 10000 curr count: 1063
2021/12/14 22:34:02 Waiting for dst bucket item count to get to: 10000 curr count: 5044
2021/12/14 22:34:07 Waiting for dst bucket item count to get to: 10000 curr count: 5044
2021/12/14 22:34:12 Waiting for dst bucket item count to get to: 10000 curr count: 9057
2021/12/14 22:34:17 Waiting for dst bucket item count to get to: 10000 curr count: 9734
2021/12/14 22:34:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 22:34:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:34:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:34:22 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 22:34:27 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 22:34:32 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 22:34:37 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 22:34:37 App: TestN1QLThrowStatement got un-deployed
2021/12/14 22:34:37 No eventing-con process running
2021/12/14 22:34:37 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/14 22:34:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:34:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 22:34:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 22:34:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:34:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:35:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 22:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.18s)
=== RUN   TestN1QLNestedForLoop
2021/12/14 22:35:11 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 22:35:11 App: TestN1QLNestedForLoop got un-deployed
2021/12/14 22:35:11 No eventing-con process running
2021/12/14 22:35:11 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/14 22:35:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:35:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:35:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:35:25 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:35:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/14 22:35:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 22:35:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 22:35:41 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 22:35:46 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 22:35:46 App: TestN1QLNestedForLoop got deployed
2021/12/14 22:35:46 Starting bucket ops to source bucket
2021/12/14 22:35:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 22:36:00 Waiting for dst bucket item count to get to: 5000 curr count: 166
2021/12/14 22:36:05 Waiting for dst bucket item count to get to: 5000 curr count: 166
2021/12/14 22:36:10 Waiting for dst bucket item count to get to: 5000 curr count: 2225
2021/12/14 22:36:15 Waiting for dst bucket item count to get to: 5000 curr count: 2225
2021/12/14 22:36:20 Waiting for dst bucket item count to get to: 5000 curr count: 4597
2021/12/14 22:36:25 Waiting for dst bucket item count to get to: 5000 curr count: 4597
2021/12/14 22:36:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 22:36:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:36:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:36:30 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 22:36:35 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 22:36:40 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 22:36:45 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 22:36:45 App: TestN1QLNestedForLoop got un-deployed
2021/12/14 22:36:45 No eventing-con process running
2021/12/14 22:36:46 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/14 22:36:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:36:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 22:36:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:36:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:37:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 22:37:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 22:37:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.08s)
=== RUN   TestN1QLAutoClose
2021/12/14 22:37:19 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 22:37:19 App: TestN1QLAutoClose got un-deployed
2021/12/14 22:37:19 No eventing-con process running
2021/12/14 22:37:19 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/14 22:37:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:37:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:37:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:37:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:37:33 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:37:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/14 22:37:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 22:37:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 22:37:49 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 22:37:54 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 22:37:54 App: TestN1QLAutoClose got deployed
2021/12/14 22:37:54 Starting bucket ops to source bucket
2021/12/14 22:37:59 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/14 22:38:04 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/14 22:38:09 Waiting for dst bucket item count to get to: 500 curr count: 9
2021/12/14 22:38:14 Waiting for dst bucket item count to get to: 500 curr count: 18
2021/12/14 22:38:19 Waiting for dst bucket item count to get to: 500 curr count: 30
2021/12/14 22:38:24 Waiting for dst bucket item count to get to: 500 curr count: 41
2021/12/14 22:38:29 Waiting for dst bucket item count to get to: 500 curr count: 52
2021/12/14 22:38:34 Waiting for dst bucket item count to get to: 500 curr count: 62
2021/12/14 22:38:39 Waiting for dst bucket item count to get to: 500 curr count: 74
2021/12/14 22:38:44 Waiting for dst bucket item count to get to: 500 curr count: 85
2021/12/14 22:38:49 Waiting for dst bucket item count to get to: 500 curr count: 95
2021/12/14 22:38:54 Waiting for dst bucket item count to get to: 500 curr count: 104
2021/12/14 22:39:00 Waiting for dst bucket item count to get to: 500 curr count: 104
2021/12/14 22:39:05 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/14 22:39:10 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/14 22:39:15 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/14 22:39:20 Waiting for dst bucket item count to get to: 500 curr count: 160
2021/12/14 22:39:25 Waiting for dst bucket item count to get to: 500 curr count: 160
2021/12/14 22:39:30 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/14 22:39:35 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/14 22:39:40 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/14 22:39:45 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/14 22:39:50 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/14 22:39:55 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/14 22:40:00 Waiting for dst bucket item count to get to: 500 curr count: 244
2021/12/14 22:40:05 Waiting for dst bucket item count to get to: 500 curr count: 244
2021/12/14 22:40:10 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/14 22:40:15 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/14 22:40:20 Waiting for dst bucket item count to get to: 500 curr count: 286
2021/12/14 22:40:25 Waiting for dst bucket item count to get to: 500 curr count: 286
2021/12/14 22:40:30 Waiting for dst bucket item count to get to: 500 curr count: 308
2021/12/14 22:40:35 Waiting for dst bucket item count to get to: 500 curr count: 308
2021/12/14 22:40:40 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/14 22:40:45 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/14 22:40:50 Waiting for dst bucket item count to get to: 500 curr count: 351
2021/12/14 22:40:55 Waiting for dst bucket item count to get to: 500 curr count: 351
2021/12/14 22:41:00 Waiting for dst bucket item count to get to: 500 curr count: 370
2021/12/14 22:41:05 Waiting for dst bucket item count to get to: 500 curr count: 370
2021/12/14 22:41:10 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/14 22:41:15 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/14 22:41:20 Waiting for dst bucket item count to get to: 500 curr count: 414
2021/12/14 22:41:25 Waiting for dst bucket item count to get to: 500 curr count: 414
2021/12/14 22:41:30 Waiting for dst bucket item count to get to: 500 curr count: 435
2021/12/14 22:41:35 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/14 22:41:40 Waiting for dst bucket item count to get to: 500 curr count: 457
2021/12/14 22:41:45 Waiting for dst bucket item count to get to: 500 curr count: 468
2021/12/14 22:41:50 Waiting for dst bucket item count to get to: 500 curr count: 479
2021/12/14 22:41:55 Waiting for dst bucket item count to get to: 500 curr count: 488
2021/12/14 22:42:00 Waiting for dst bucket item count to get to: 500 curr count: 498
2021/12/14 22:42:00 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/14 22:42:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:42:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:42:05 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 22:42:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 22:42:15 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 22:42:20 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 22:42:20 App: TestN1QLAutoClose got un-deployed
2021/12/14 22:42:20 No eventing-con process running
2021/12/14 22:42:20 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/14 22:42:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:42:32 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:42:37 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 22:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:42:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:42:49 Waiting for dst bucket item count to get to: 0 curr count: 432
2021/12/14 22:42:54 Waiting for dst bucket item count to get to: 0 curr count: 182
2021/12/14 22:42:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.44s)
=== RUN   TestN1QLPosParams
2021/12/14 22:42:59 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 22:42:59 App: TestN1QLPosParams got un-deployed
2021/12/14 22:42:59 No eventing-con process running
2021/12/14 22:42:59 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/14 22:43:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:43:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:43:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:43:12 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:43:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/14 22:43:18 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 22:43:23 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 22:43:28 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 22:43:33 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 22:43:33 App: TestN1QLPosParams got deployed
2021/12/14 22:43:33 Starting bucket ops to source bucket
2021/12/14 22:43:43 Waiting for dst bucket item count to get to: 5000 curr count: 620
2021/12/14 22:43:48 Waiting for dst bucket item count to get to: 5000 curr count: 620
2021/12/14 22:43:53 Waiting for dst bucket item count to get to: 5000 curr count: 1924
2021/12/14 22:43:58 Waiting for dst bucket item count to get to: 5000 curr count: 3804
2021/12/14 22:43:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 22:43:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:43:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:44:03 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 22:44:08 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 22:44:13 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 22:44:18 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 22:44:18 App: TestN1QLPosParams got un-deployed
2021/12/14 22:44:18 No eventing-con process running
2021/12/14 22:44:18 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/14 22:44:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:44:30 Waiting for dst bucket item count to get to: 0 curr count: 2982
2021/12/14 22:44:35 Waiting for dst bucket item count to get to: 0 curr count: 2982
2021/12/14 22:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:44:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 22:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.12s)
=== RUN   TestN1QLExhaustConnPool
2021/12/14 22:44:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 22:44:52 App: TestN1QLExhaustConnPool got un-deployed
2021/12/14 22:44:52 No eventing-con process running
2021/12/14 22:44:52 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/14 22:44:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:44:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:45:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:45:05 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:45:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/14 22:45:11 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 22:45:16 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 22:45:21 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 22:45:26 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 22:45:26 App: TestN1QLExhaustConnPool got deployed
2021/12/14 22:45:26 Starting bucket ops to source bucket
2021/12/14 22:45:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 22:45:37 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 22:45:42 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 22:45:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 22:45:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:45:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:45:47 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 22:45:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 22:45:57 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 22:46:02 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 22:46:02 App: TestN1QLExhaustConnPool got un-deployed
2021/12/14 22:46:02 No eventing-con process running
2021/12/14 22:46:02 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/14 22:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:46:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 22:46:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 22:46:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:46:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:46:31 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/14 22:46:36 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/14 22:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.86s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/14 22:46:41 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 22:46:41 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/14 22:46:41 No eventing-con process running
2021/12/14 22:46:41 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/14 22:46:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:46:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:46:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:46:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:46:54 Starting bucket ops to source bucket
2021/12/14 22:47:12 Starting bucket ops to source bucket
2021/12/14 22:47:29 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 11089
2021/12/14 22:47:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/14 22:47:36 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 22:47:41 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 22:47:46 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 22:47:51 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 22:47:51 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/14 22:47:51 Starting bucket ops to source bucket
2021/12/14 22:47:56 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:01 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:06 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:11 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:16 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:21 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:26 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:31 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:36 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:41 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 22:48:46 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/14 22:48:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:48:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:48:51 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 22:48:56 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 22:49:01 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 22:49:06 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 22:49:06 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/14 22:49:06 No eventing-con process running
2021/12/14 22:49:06 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/14 22:49:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:49:18 Waiting for dst bucket item count to get to: 0 curr count: 18828
2021/12/14 22:49:23 Waiting for dst bucket item count to get to: 0 curr count: 3828
2021/12/14 22:49:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:49:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:49:35 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/14 22:49:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.14s)
=== RUN   TestN1QLQueryConsistency
2021/12/14 22:49:40 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 22:49:40 App: n1qlquery_consistency got un-deployed
2021/12/14 22:49:40 No eventing-con process running
2021/12/14 22:49:40 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/14 22:49:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:49:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:49:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:49:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:49:53 Starting bucket ops to source bucket
2021/12/14 22:49:56 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:49:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/14 22:50:02 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 22:50:07 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 22:50:12 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 22:50:18 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 22:50:18 App: n1qlquery_consistency got deployed
2021/12/14 22:50:23 Waiting for dst bucket item count to get to: 5000 curr count: 56
2021/12/14 22:50:28 Waiting for dst bucket item count to get to: 5000 curr count: 2507
2021/12/14 22:50:33 Waiting for dst bucket item count to get to: 5000 curr count: 4951
2021/12/14 22:50:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 22:50:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 22:50:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:50:38 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 22:50:43 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 22:50:48 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 22:50:53 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 22:50:53 App: n1qlquery_consistency got un-deployed
2021/12/14 22:50:53 No eventing-con process running
2021/12/14 22:50:53 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/14 22:50:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:51:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 22:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 22:51:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 22:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.68s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1569.021s

N1ql tests took 0 hours, 26 minutes and 13 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/14 22:52:51 Started rebalance
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
=== RUN   TestDuplicateWithEventingRebIn
2021/12/14 22:53:10 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/14 22:53:15 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 22:53:15 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/14 22:53:15 No eventing-con process running
2021/12/14 22:53:15 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/14 22:53:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:53:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:53:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:53:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:53:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:53:33 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:53:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 22:53:40 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 22:53:45 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 22:53:50 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 22:53:55 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 22:53:55 App: TestDuplicateWithEventingRebIn got deployed
2021/12/14 22:53:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 22:53:55 Starting bucket ops to source bucket
2021/12/14 22:54:09 Removing node(s): [] from the cluster
2021/12/14 22:54:14 Rebalance progress: 0
2021/12/14 22:54:19 Rebalance progress: 0
2021/12/14 22:54:24 Rebalance progress: 25
2021/12/14 22:54:29 Rebalance progress: 25
2021/12/14 22:54:34 Rebalance progress: 25
2021/12/14 22:54:39 Rebalance progress: 98.58490566037736
2021/12/14 22:54:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:54:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:54:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 22:54:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 22:54:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 22:54:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 22:54:50 Rebalance progress: 25
2021/12/14 22:54:55 Rebalance progress: 25
2021/12/14 22:55:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:55:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:55:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 22:55:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 22:55:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 22:55:00 Waiting for src bucket item count to get to: 200000 curr count: 87227
2021/12/14 22:55:05 Waiting for src bucket item count to get to: 200000 curr count: 103775
2021/12/14 22:55:10 Waiting for src bucket item count to get to: 200000 curr count: 103775
2021/12/14 22:55:15 Waiting for src bucket item count to get to: 200000 curr count: 120070
2021/12/14 22:55:20 Waiting for src bucket item count to get to: 200000 curr count: 120070
2021/12/14 22:55:25 Waiting for src bucket item count to get to: 200000 curr count: 136599
2021/12/14 22:55:30 Waiting for src bucket item count to get to: 200000 curr count: 136599
2021/12/14 22:55:35 Waiting for src bucket item count to get to: 200000 curr count: 153186
2021/12/14 22:55:40 Waiting for src bucket item count to get to: 200000 curr count: 153186
2021/12/14 22:55:45 Waiting for src bucket item count to get to: 200000 curr count: 169638
2021/12/14 22:55:50 Waiting for src bucket item count to get to: 200000 curr count: 169638
2021/12/14 22:55:55 Waiting for src bucket item count to get to: 200000 curr count: 185874
2021/12/14 22:56:00 Waiting for src bucket item count to get to: 200000 curr count: 185874
2021/12/14 22:56:10 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:15 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:20 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:25 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:30 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:35 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:40 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:45 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:50 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:55 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 22:56:55 Post SET ops. Expected item count: 200000 got 200001
2021/12/14 22:56:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 22:57:00 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 22:57:05 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 22:57:10 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 22:57:15 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 22:57:15 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/14 22:57:15 No eventing-con process running
2021/12/14 22:57:15 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/14 22:57:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:57:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:57:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 22:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:57:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:57:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:57:38 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/14 22:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (278.30s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/14 22:57:43 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 22:57:44 Removing node(s): [] from the cluster
2021/12/14 22:57:49 Rebalance progress: 100
2021/12/14 22:57:54 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 22:57:54 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/14 22:57:54 No eventing-con process running
2021/12/14 22:57:54 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/14 22:57:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:58:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:58:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 22:58:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:58:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:58:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 22:58:12 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 22:58:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 22:58:18 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 22:58:23 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 22:58:28 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 22:58:33 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 22:58:33 App: TestDuplicateWithEventingRebOut got deployed
2021/12/14 22:58:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 22:58:33 Starting bucket ops to source bucket
2021/12/14 22:58:38 Rebalance progress: 25
2021/12/14 22:58:43 Rebalance progress: 25
2021/12/14 22:58:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:58:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 22:58:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 22:58:48 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 22:58:48 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 22:58:48 Waiting for src bucket item count to get to: 200000 curr count: 9375
2021/12/14 22:58:53 Waiting for src bucket item count to get to: 200000 curr count: 25935
2021/12/14 22:58:58 Waiting for src bucket item count to get to: 200000 curr count: 25935
2021/12/14 22:59:03 Waiting for src bucket item count to get to: 200000 curr count: 42330
2021/12/14 22:59:08 Waiting for src bucket item count to get to: 200000 curr count: 42330
2021/12/14 22:59:13 Waiting for src bucket item count to get to: 200000 curr count: 58718
2021/12/14 22:59:18 Waiting for src bucket item count to get to: 200000 curr count: 58718
2021/12/14 22:59:23 Waiting for src bucket item count to get to: 200000 curr count: 58718
2021/12/14 22:59:28 Waiting for src bucket item count to get to: 200000 curr count: 74981
2021/12/14 22:59:33 Waiting for src bucket item count to get to: 200000 curr count: 74981
2021/12/14 22:59:38 Waiting for src bucket item count to get to: 200000 curr count: 91004
2021/12/14 22:59:43 Waiting for src bucket item count to get to: 200000 curr count: 91004
2021/12/14 22:59:48 Waiting for src bucket item count to get to: 200000 curr count: 107373
2021/12/14 22:59:53 Waiting for src bucket item count to get to: 200000 curr count: 107373
2021/12/14 22:59:58 Waiting for src bucket item count to get to: 200000 curr count: 123697
2021/12/14 23:00:03 Waiting for src bucket item count to get to: 200000 curr count: 123697
2021/12/14 23:00:08 Waiting for src bucket item count to get to: 200000 curr count: 140266
2021/12/14 23:00:13 Waiting for src bucket item count to get to: 200000 curr count: 140266
2021/12/14 23:00:18 Waiting for src bucket item count to get to: 200000 curr count: 156586
2021/12/14 23:00:23 Waiting for src bucket item count to get to: 200000 curr count: 156586
2021/12/14 23:00:28 Waiting for src bucket item count to get to: 200000 curr count: 172914
2021/12/14 23:00:33 Waiting for src bucket item count to get to: 200000 curr count: 172914
2021/12/14 23:00:38 Waiting for src bucket item count to get to: 200000 curr count: 189004
2021/12/14 23:00:43 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 23:00:43 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 23:00:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:00:49 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 23:00:54 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 23:00:59 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 23:01:04 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 23:01:04 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/14 23:01:04 No eventing-con process running
2021/12/14 23:01:04 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/14 23:01:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:01:15 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:01:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:01:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:01:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:01:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.82s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/14 23:01:37 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 23:01:37 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/14 23:01:37 No eventing-con process running
2021/12/14 23:01:37 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/14 23:01:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:01:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:01:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:01:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:01:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:01:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:01:56 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:01:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 23:02:02 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 23:02:07 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 23:02:12 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 23:02:17 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 23:02:17 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/14 23:02:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:02:17 Starting bucket ops to source bucket
2021/12/14 23:02:17 Removing node(s): [] from the cluster
2021/12/14 23:02:22 Rebalance progress: 25
2021/12/14 23:02:27 Rebalance progress: 25
2021/12/14 23:02:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:02:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 23:02:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:02:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:02:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:02:44 Rebalance progress: 16.66666666666667
2021/12/14 23:02:49 Rebalance progress: 16.66666666666667
2021/12/14 23:02:54 Rebalance progress: 16.66666666666667
2021/12/14 23:02:59 Rebalance progress: 95.30905077262692
2021/12/14 23:03:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:03:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:03:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:03:09 Rebalance progress: 16.66666666666667
2021/12/14 23:03:14 Rebalance progress: 16.66666666666667
2021/12/14 23:03:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:03:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:03:19 Waiting for src bucket item count to get to: 200000 curr count: 78475
2021/12/14 23:03:25 Waiting for src bucket item count to get to: 200000 curr count: 94549
2021/12/14 23:03:30 Waiting for src bucket item count to get to: 200000 curr count: 94549
2021/12/14 23:03:35 Waiting for src bucket item count to get to: 200000 curr count: 111037
2021/12/14 23:03:40 Waiting for src bucket item count to get to: 200000 curr count: 111037
2021/12/14 23:03:45 Waiting for src bucket item count to get to: 200000 curr count: 127505
2021/12/14 23:03:50 Waiting for src bucket item count to get to: 200000 curr count: 127505
2021/12/14 23:03:55 Waiting for src bucket item count to get to: 200000 curr count: 143587
2021/12/14 23:04:00 Waiting for src bucket item count to get to: 200000 curr count: 143587
2021/12/14 23:04:05 Waiting for src bucket item count to get to: 200000 curr count: 159777
2021/12/14 23:04:10 Waiting for src bucket item count to get to: 200000 curr count: 159777
2021/12/14 23:04:15 Waiting for src bucket item count to get to: 200000 curr count: 175609
2021/12/14 23:04:20 Waiting for src bucket item count to get to: 200000 curr count: 175609
2021/12/14 23:04:25 Waiting for src bucket item count to get to: 200000 curr count: 191870
2021/12/14 23:04:30 Waiting for src bucket item count to get to: 200000 curr count: 191870
2021/12/14 23:04:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 23:04:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 23:04:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:04:40 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 23:04:45 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 23:04:50 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 23:04:55 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 23:04:55 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/14 23:04:55 No eventing-con process running
2021/12/14 23:04:55 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/14 23:04:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:05:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:05:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:05:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:05:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:05:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:05:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.11s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/14 23:05:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 23:05:28 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/14 23:05:28 No eventing-con process running
2021/12/14 23:05:28 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/14 23:05:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:05:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:05:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:05:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:05:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:05:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:05:47 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:05:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 23:05:53 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 23:05:58 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 23:06:03 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 23:06:08 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 23:06:08 App: TestDuplicateWithKVRebIn got deployed
2021/12/14 23:06:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 23:06:08 Starting bucket ops to source bucket
2021/12/14 23:06:08 Removing node(s): [] from the cluster
2021/12/14 23:06:13 Rebalance progress: 16.66666666666667
2021/12/14 23:06:18 Rebalance progress: 16.66666666666667
2021/12/14 23:06:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:06:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:06:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:06:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:06:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:06:28 Rebalance progress: 16.66666666666667
2021/12/14 23:06:33 Rebalance progress: 16.66666666666667
2021/12/14 23:06:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:06:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:06:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:06:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:06:38 Waiting for src bucket item count to get to: 200000 curr count: 33050
2021/12/14 23:06:43 Waiting for src bucket item count to get to: 200000 curr count: 49278
2021/12/14 23:06:48 Waiting for src bucket item count to get to: 200000 curr count: 49278
2021/12/14 23:06:53 Waiting for src bucket item count to get to: 200000 curr count: 65727
2021/12/14 23:06:58 Waiting for src bucket item count to get to: 200000 curr count: 65727
2021/12/14 23:07:03 Waiting for src bucket item count to get to: 200000 curr count: 81297
2021/12/14 23:07:08 Waiting for src bucket item count to get to: 200000 curr count: 81297
2021/12/14 23:07:13 Waiting for src bucket item count to get to: 200000 curr count: 81297
2021/12/14 23:07:18 Waiting for src bucket item count to get to: 200000 curr count: 97329
2021/12/14 23:07:23 Waiting for src bucket item count to get to: 200000 curr count: 97329
2021/12/14 23:07:29 Waiting for src bucket item count to get to: 200000 curr count: 113731
2021/12/14 23:07:34 Waiting for src bucket item count to get to: 200000 curr count: 113731
2021/12/14 23:07:39 Waiting for src bucket item count to get to: 200000 curr count: 130219
2021/12/14 23:07:44 Waiting for src bucket item count to get to: 200000 curr count: 130219
2021/12/14 23:07:49 Waiting for src bucket item count to get to: 200000 curr count: 146749
2021/12/14 23:07:54 Waiting for src bucket item count to get to: 200000 curr count: 146749
2021/12/14 23:07:59 Waiting for src bucket item count to get to: 200000 curr count: 163125
2021/12/14 23:08:04 Waiting for src bucket item count to get to: 200000 curr count: 163125
2021/12/14 23:08:09 Waiting for src bucket item count to get to: 200000 curr count: 178697
2021/12/14 23:08:14 Waiting for src bucket item count to get to: 200000 curr count: 178697
2021/12/14 23:08:19 Waiting for src bucket item count to get to: 200000 curr count: 194705
2021/12/14 23:08:24 Waiting for src bucket item count to get to: 200000 curr count: 194705
2021/12/14 23:08:29 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 23:08:29 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 23:08:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:08:34 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 23:08:39 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 23:08:44 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 23:08:49 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 23:08:49 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/14 23:08:49 No eventing-con process running
2021/12/14 23:08:49 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/14 23:08:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:09:01 Waiting for dst bucket item count to get to: 0 curr count: 112900
2021/12/14 23:09:06 Waiting for dst bucket item count to get to: 0 curr count: 112900
2021/12/14 23:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:09:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:09:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:09:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:09:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.15s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/14 23:09:22 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 23:09:23 Removing node(s): [] from the cluster
2021/12/14 23:09:28 Rebalance progress: 100
2021/12/14 23:09:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 23:09:33 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/14 23:09:33 No eventing-con process running
2021/12/14 23:09:33 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/14 23:09:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:09:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:09:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:09:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:09:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:09:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:09:51 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:09:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 23:09:57 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 23:10:03 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 23:10:08 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 23:10:13 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 23:10:13 App: TestDuplicateWithKVRebOut got deployed
2021/12/14 23:10:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:10:13 Starting bucket ops to source bucket
2021/12/14 23:10:18 Rebalance progress: 16.66666666666667
2021/12/14 23:10:23 Rebalance progress: 16.66666666666667
2021/12/14 23:10:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:10:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:10:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:10:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:10:28 Waiting for src bucket item count to get to: 200000 curr count: 10187
2021/12/14 23:10:33 Waiting for src bucket item count to get to: 200000 curr count: 26149
2021/12/14 23:10:38 Waiting for src bucket item count to get to: 200000 curr count: 26149
2021/12/14 23:10:43 Waiting for src bucket item count to get to: 200000 curr count: 42461
2021/12/14 23:10:48 Waiting for src bucket item count to get to: 200000 curr count: 42461
2021/12/14 23:10:53 Waiting for src bucket item count to get to: 200000 curr count: 58949
2021/12/14 23:10:58 Waiting for src bucket item count to get to: 200000 curr count: 58949
2021/12/14 23:11:03 Waiting for src bucket item count to get to: 200000 curr count: 58949
2021/12/14 23:11:08 Waiting for src bucket item count to get to: 200000 curr count: 74664
2021/12/14 23:11:13 Waiting for src bucket item count to get to: 200000 curr count: 74664
2021/12/14 23:11:18 Waiting for src bucket item count to get to: 200000 curr count: 90836
2021/12/14 23:11:23 Waiting for src bucket item count to get to: 200000 curr count: 90836
2021/12/14 23:11:28 Waiting for src bucket item count to get to: 200000 curr count: 107127
2021/12/14 23:11:33 Waiting for src bucket item count to get to: 200000 curr count: 107127
2021/12/14 23:11:38 Waiting for src bucket item count to get to: 200000 curr count: 123089
2021/12/14 23:11:43 Waiting for src bucket item count to get to: 200000 curr count: 123089
2021/12/14 23:11:48 Waiting for src bucket item count to get to: 200000 curr count: 139089
2021/12/14 23:11:53 Waiting for src bucket item count to get to: 200000 curr count: 139089
2021/12/14 23:11:58 Waiting for src bucket item count to get to: 200000 curr count: 155312
2021/12/14 23:12:03 Waiting for src bucket item count to get to: 200000 curr count: 155312
2021/12/14 23:12:08 Waiting for src bucket item count to get to: 200000 curr count: 171119
2021/12/14 23:12:13 Waiting for src bucket item count to get to: 200000 curr count: 171119
2021/12/14 23:12:18 Waiting for src bucket item count to get to: 200000 curr count: 187056
2021/12/14 23:12:23 Waiting for src bucket item count to get to: 200000 curr count: 187056
2021/12/14 23:12:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 23:12:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 23:12:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:12:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 23:12:38 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 23:12:43 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 23:12:48 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 23:12:48 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/14 23:12:48 No eventing-con process running
2021/12/14 23:12:48 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/14 23:12:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:12:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:13:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:13:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.12s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/14 23:13:17 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 23:13:17 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/14 23:13:17 No eventing-con process running
2021/12/14 23:13:17 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/14 23:13:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:13:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:13:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:13:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:13:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:13:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:13:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:13:35 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:13:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 23:13:41 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 23:13:46 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 23:13:51 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 23:13:56 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 23:13:56 App: TestDuplicateWithKVSwapReb got deployed
2021/12/14 23:13:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 23:13:56 Starting bucket ops to source bucket
2021/12/14 23:13:57 Removing node(s): [] from the cluster
2021/12/14 23:14:02 Rebalance progress: 16.66666666666667
2021/12/14 23:14:07 Rebalance progress: 16.66666666666667
2021/12/14 23:14:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:14:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:14:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:14:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:14:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 23:14:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:14:18 Rebalance progress: 16.66666666666667
2021/12/14 23:14:23 Rebalance progress: 16.66666666666667
2021/12/14 23:14:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:14:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:14:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:14:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:14:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:14:33 Rebalance progress: 16.66666666666667
2021/12/14 23:14:38 Rebalance progress: 16.66666666666667
2021/12/14 23:14:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:14:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:14:43 Waiting for src bucket item count to get to: 200000 curr count: 67946
2021/12/14 23:14:48 Waiting for src bucket item count to get to: 200000 curr count: 67946
2021/12/14 23:14:53 Waiting for src bucket item count to get to: 200000 curr count: 83085
2021/12/14 23:14:58 Waiting for src bucket item count to get to: 200000 curr count: 83085
2021/12/14 23:15:03 Waiting for src bucket item count to get to: 200000 curr count: 99030
2021/12/14 23:15:08 Waiting for src bucket item count to get to: 200000 curr count: 99030
2021/12/14 23:15:13 Waiting for src bucket item count to get to: 200000 curr count: 114937
2021/12/14 23:15:18 Waiting for src bucket item count to get to: 200000 curr count: 114937
2021/12/14 23:15:23 Waiting for src bucket item count to get to: 200000 curr count: 131025
2021/12/14 23:15:28 Waiting for src bucket item count to get to: 200000 curr count: 131025
2021/12/14 23:15:33 Waiting for src bucket item count to get to: 200000 curr count: 147444
2021/12/14 23:15:38 Waiting for src bucket item count to get to: 200000 curr count: 147444
2021/12/14 23:15:43 Waiting for src bucket item count to get to: 200000 curr count: 163891
2021/12/14 23:15:48 Waiting for src bucket item count to get to: 200000 curr count: 163891
2021/12/14 23:15:53 Waiting for src bucket item count to get to: 200000 curr count: 179357
2021/12/14 23:15:58 Waiting for src bucket item count to get to: 200000 curr count: 179357
2021/12/14 23:16:03 Waiting for src bucket item count to get to: 200000 curr count: 195595
2021/12/14 23:16:08 Waiting for src bucket item count to get to: 200000 curr count: 195595
2021/12/14 23:16:13 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 23:16:13 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 23:16:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:16:18 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 23:16:23 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 23:16:28 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 23:16:33 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 23:16:33 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/14 23:16:33 No eventing-con process running
2021/12/14 23:16:33 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/14 23:16:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:16:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:16:45 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:16:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:16:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:16:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 23:16:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.99s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/14 23:17:06 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:17:07 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 23:17:07 No eventing-con process running
2021/12/14 23:17:07 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 23:17:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:17:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:17:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:17:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:17:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:17:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:17:25 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:17:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 23:17:31 Starting bucket ops to source bucket
2021/12/14 23:17:36 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:17:41 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:17:46 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:17:46 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:17:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:17:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:17:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:17:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:17:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:17:52 Rebalance progress: 16.66666666666667
2021/12/14 23:17:57 Rebalance progress: 16.66666666666667
2021/12/14 23:18:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:18:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:18:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:18:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:18:07 Rebalance progress: 16.66666666666667
2021/12/14 23:18:12 Rebalance progress: 16.66666666666667
2021/12/14 23:18:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:18:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 23:18:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 23:18:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:18:28 Rebalance progress: 12.5
2021/12/14 23:18:33 Rebalance progress: 12.5
2021/12/14 23:18:38 Rebalance progress: 12.5
2021/12/14 23:18:43 Rebalance progress: 57.3
2021/12/14 23:18:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:18:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:18:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:18:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:18:53 Rebalance progress: 12.5
2021/12/14 23:18:58 Rebalance progress: 12.5
2021/12/14 23:19:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:19:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:19:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:19:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:19:08 Rebalance progress: 12.5
2021/12/14 23:19:13 Rebalance progress: 12.5
2021/12/14 23:19:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:19:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:19:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:19:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 23:19:23 Rebalance progress: 12.5
2021/12/14 23:19:28 Rebalance progress: 12.5
2021/12/14 23:19:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:19:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:19:38 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:19:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:19:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:19:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:19:53 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 23:19:53 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:19:53 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 23:19:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:20:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:20:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12195
2021/12/14 23:20:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:20:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:20:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:20:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12195
2021/12/14 23:20:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.58s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/14 23:20:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 23:20:27 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/14 23:20:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:20:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/14 23:20:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:20:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:20:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:20:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:20:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:20:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:20:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:20:46 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:20:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/14 23:20:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:20:57 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 23:21:02 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 23:21:07 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 23:21:07 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:21:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:21:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:21:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:21:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:21:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 23:21:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:21:13 Rebalance progress: 12.5
2021/12/14 23:21:18 Rebalance progress: 12.5
2021/12/14 23:21:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:21:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:21:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:21:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 23:21:28 Rebalance progress: 12.5
2021/12/14 23:21:33 Rebalance progress: 12.5
2021/12/14 23:21:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:21:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:21:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:21:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:21:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 23:21:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 23:21:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 23:21:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 23:21:58 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/14 23:21:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:21:58 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/14 23:22:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:22:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:22:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:22:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:22:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:22:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4586
2021/12/14 23:22:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.50s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/14 23:22:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 23:22:27 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 23:22:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:22:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/14 23:22:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:22:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:22:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:22:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:22:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:22:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:22:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:22:45 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:22:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 23:22:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:22:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 23:23:01 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 23:23:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 23:23:06 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:23:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:23:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:23:12 Rebalance progress: 12.5
2021/12/14 23:23:17 Rebalance progress: 12.5
2021/12/14 23:23:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:23:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:23:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:23:27 Rebalance progress: 12.5
2021/12/14 23:23:32 Rebalance progress: 12.5
2021/12/14 23:23:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:23:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 23:23:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:23:43 Rebalance progress: 12.5
2021/12/14 23:23:48 Rebalance progress: 12.5
2021/12/14 23:23:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:23:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:23:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:23:58 Rebalance progress: 12.5
2021/12/14 23:24:03 Rebalance progress: 12.5
2021/12/14 23:24:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:24:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:24:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:24:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:24:13 Rebalance progress: 12.5
2021/12/14 23:24:18 Rebalance progress: 12.5
2021/12/14 23:24:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:24:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:24:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:24:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 23:24:28 Rebalance progress: 12.5
2021/12/14 23:24:33 Rebalance progress: 12.5
2021/12/14 23:24:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:24:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:24:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:24:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:24:43 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 23:24:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 23:24:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 23:24:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 23:24:58 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 23:24:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:24:58 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/14 23:25:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:25:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:25:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 278
2021/12/14 23:25:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 278
2021/12/14 23:25:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:25:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:25:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:25:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10655
2021/12/14 23:25:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.12s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/14 23:25:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 23:25:37 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/14 23:25:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:25:37 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/14 23:25:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:25:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:25:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:25:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:25:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:25:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:25:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:25:50 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:25:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 23:25:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:26:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 23:26:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 23:26:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 23:26:11 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:26:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:26:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:26:17 Rebalance progress: 12.5
2021/12/14 23:26:22 Rebalance progress: 12.5
2021/12/14 23:26:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:26:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:26:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:26:32 Rebalance progress: 12.5
2021/12/14 23:26:37 Rebalance progress: 12.5
2021/12/14 23:26:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:26:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:26:47 Rebalance progress: 12.5
2021/12/14 23:26:52 Rebalance progress: 12.5
2021/12/14 23:26:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:26:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:27:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 23:27:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 23:27:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 23:27:17 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 23:27:17 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/14 23:27:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:27:17 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/14 23:27:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:27:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:27:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6056
2021/12/14 23:27:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:27:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:27:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:27:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4957
2021/12/14 23:27:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4957
2021/12/14 23:27:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.12s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/14 23:27:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:27:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:27:56 Rebalance progress: 100
2021/12/14 23:28:01 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 23:28:01 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/14 23:28:01 No eventing-con process running
2021/12/14 23:28:01 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/14 23:28:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:28:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:28:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:28:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:28:15 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:28:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 23:28:21 Starting bucket ops to source bucket
2021/12/14 23:28:26 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 23:28:31 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 23:28:36 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 23:28:36 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:28:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:28:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:28:42 Rebalance progress: 12.5
2021/12/14 23:28:47 Rebalance progress: 12.5
2021/12/14 23:28:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:28:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:28:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 23:28:52 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 23:28:57 Rebalance progress: 12.5
2021/12/14 23:29:02 Rebalance progress: 12.5
2021/12/14 23:29:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:29:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:29:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:29:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:29:12 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 23:29:17 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 23:29:22 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 23:29:27 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 23:29:27 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/14 23:29:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:29:27 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/14 23:29:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:29:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:29:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4568
2021/12/14 23:29:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:29:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:29:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:29:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3111
2021/12/14 23:29:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3111
2021/12/14 23:29:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.84s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/14 23:30:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:30:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:30:06 Rebalance progress: 100
2021/12/14 23:30:11 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 23:30:11 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/14 23:30:11 No eventing-con process running
2021/12/14 23:30:11 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/14 23:30:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:30:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:30:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:30:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:30:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:30:25 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:30:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 23:30:31 Starting bucket ops to source bucket
2021/12/14 23:30:36 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 23:30:41 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 23:30:46 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 23:30:46 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:30:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:30:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:30:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:30:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:30:51 Rebalance progress: 12.5
2021/12/14 23:30:56 Rebalance progress: 12.5
2021/12/14 23:31:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:31:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:31:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:31:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 23:31:01 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 23:31:06 Rebalance progress: 12.5
2021/12/14 23:31:11 Rebalance progress: 12.5
2021/12/14 23:31:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:31:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:31:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:31:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:31:16 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 23:31:21 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 23:31:26 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 23:31:31 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 23:31:37 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 23:31:37 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/14 23:31:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:31:37 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/14 23:31:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:31:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:31:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37745
2021/12/14 23:31:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:31:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:31:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:32:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8930
2021/12/14 23:32:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8930
2021/12/14 23:32:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.49s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/14 23:32:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 23:32:15 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/14 23:32:15 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:32:15 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/14 23:32:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:32:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:32:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:32:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:32:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:32:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:32:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:32:34 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:32:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:32:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:32:35 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:32:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 23:32:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:32:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 23:32:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 23:32:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 23:32:56 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/14 23:33:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 23:33:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 23:33:11 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 23:33:11 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:33:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:33:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:33:16 Rebalance progress: 12.5
2021/12/14 23:33:21 Rebalance progress: 12.5
2021/12/14 23:33:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:33:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:33:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:33:32 Rebalance progress: 12.5
2021/12/14 23:33:37 Rebalance progress: 12.5
2021/12/14 23:33:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:33:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 23:33:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:33:47 Rebalance progress: 12.5
2021/12/14 23:33:52 Rebalance progress: 12.5
2021/12/14 23:33:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:33:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:33:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:34:03 Rebalance progress: 12.5
2021/12/14 23:34:08 Rebalance progress: 12.5
2021/12/14 23:34:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:34:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:34:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:34:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:34:18 Rebalance progress: 12.5
2021/12/14 23:34:23 Rebalance progress: 12.5
2021/12/14 23:34:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:34:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:34:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:34:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 23:34:33 Rebalance progress: 12.5
2021/12/14 23:34:38 Rebalance progress: 12.5
2021/12/14 23:34:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:34:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:34:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:34:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:34:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 23:34:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 23:34:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 23:35:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 23:35:03 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/14 23:35:03 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:35:03 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/14 23:35:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:35:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:35:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12175
2021/12/14 23:35:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:35:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:35:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:35:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12175
2021/12/14 23:35:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:35:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:35:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 23:35:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 23:35:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 23:35:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 23:35:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/14 23:35:48 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:35:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/14 23:35:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:35:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:35:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:35:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:36:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:36:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (236.54s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/14 23:36:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 23:36:12 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/14 23:36:12 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:36:12 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/14 23:36:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:36:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:36:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:36:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:36:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:36:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:36:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:36:30 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:36:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:36:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:36:31 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:36:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 23:36:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:36:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 23:36:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 23:36:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 23:36:53 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/14 23:36:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 23:37:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 23:37:08 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 23:37:08 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:37:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:37:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:37:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:37:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:37:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 23:37:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:37:14 Rebalance progress: 12.5
2021/12/14 23:37:19 Rebalance progress: 12.5
2021/12/14 23:37:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:37:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:37:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:37:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 23:37:29 Rebalance progress: 12.5
2021/12/14 23:37:34 Rebalance progress: 12.5
2021/12/14 23:37:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:37:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:37:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 23:37:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 23:37:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 23:37:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 23:37:59 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/14 23:37:59 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:37:59 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/14 23:38:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:38:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:38:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6137
2021/12/14 23:38:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6137
2021/12/14 23:38:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:38:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:38:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:38:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6137
2021/12/14 23:38:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:38:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:38:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 23:38:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 23:38:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 23:38:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 23:38:49 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/14 23:38:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:38:49 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/14 23:38:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:38:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:38:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:38:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.38s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/14 23:39:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 23:39:13 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 23:39:13 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:39:13 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/14 23:39:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:39:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:39:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 23:39:32 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 23:39:32 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:39:32 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/14 23:39:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:39:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:45 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:39:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:39:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:39:46 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:39:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 23:39:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:39:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 23:40:03 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 23:40:08 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 23:40:08 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/14 23:40:13 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 23:40:18 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 23:40:23 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 23:40:23 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:40:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:40:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:40:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:40:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:40:28 Rebalance progress: 12.5
2021/12/14 23:40:33 Rebalance progress: 12.5
2021/12/14 23:40:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:40:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:40:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:40:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:40:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:40:43 Rebalance progress: 12.5
2021/12/14 23:40:48 Rebalance progress: 12.5
2021/12/14 23:40:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:40:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:40:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:40:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:40:59 Rebalance progress: 12.5
2021/12/14 23:41:04 Rebalance progress: 12.5
2021/12/14 23:41:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:41:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:41:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:41:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:41:14 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 23:41:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 23:41:24 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 23:41:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 23:41:29 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 23:41:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:41:29 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/14 23:41:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:41:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:41:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6069
2021/12/14 23:41:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6069
2021/12/14 23:41:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:41:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:41:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:41:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:41:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:42:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 23:42:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 23:42:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 23:42:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 23:42:15 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 23:42:15 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:42:15 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/14 23:42:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:42:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:42:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:42:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:42:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:42:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (205.24s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 23:42:38 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:42:38 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 23:42:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:42:38 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 23:42:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:42:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:42:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:42:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:42:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:42:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:42:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:42:57 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:42:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 23:43:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:43:08 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:43:13 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:43:18 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:43:18 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:43:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:43:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:43:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:43:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:43:38 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:43:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:43:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:44:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:44:14 Rebalance progress: 12.5
2021/12/14 23:44:19 Rebalance progress: 12.5
2021/12/14 23:44:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:44:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:44:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:44:30 Rebalance progress: 12.5
2021/12/14 23:44:35 Rebalance progress: 12.5
2021/12/14 23:44:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:44:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:44:45 Rebalance progress: 12.5
2021/12/14 23:44:50 Rebalance progress: 12.5
2021/12/14 23:44:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:44:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:45:00 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:45:05 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:45:10 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:45:15 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:45:15 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 23:45:15 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:45:15 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 23:45:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:45:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:45:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11163
2021/12/14 23:45:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:45:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:45:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:45:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11163
2021/12/14 23:45:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (190.20s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 23:45:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:45:49 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 23:45:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:45:49 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 23:45:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:45:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:45:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:45:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:46:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:46:07 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:46:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 23:46:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:46:18 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:46:23 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:46:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 23:46:28 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:46:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:46:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:46:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:46:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:47:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:47:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:47:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:47:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:47:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:47:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:47:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:47:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:47:51 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:48:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:48:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:48:21 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:48:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:48:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:48:52 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:48:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:49:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:49:08 Rebalance progress: 12.5
2021/12/14 23:49:13 Rebalance progress: 12.5
2021/12/14 23:49:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:49:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:49:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:49:23 Rebalance progress: 12.5
2021/12/14 23:49:28 Rebalance progress: 12.5
2021/12/14 23:49:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:49:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:49:38 Rebalance progress: 12.5
2021/12/14 23:49:43 Rebalance progress: 12.5
2021/12/14 23:49:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:49:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:49:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:49:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:49:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:50:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:50:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 23:50:08 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 23:50:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:50:08 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 23:50:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:50:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:50:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5373
2021/12/14 23:50:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5373
2021/12/14 23:50:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:50:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:50:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:50:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21488
2021/12/14 23:50:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.48s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/14 23:50:47 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 23:50:47 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 23:50:47 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:50:47 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/14 23:50:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:50:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:50:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:50:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:51:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:51:06 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:51:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 23:51:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:51:17 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 23:51:22 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 23:51:27 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 23:51:27 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:51:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:51:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:51:32 Rebalance progress: 12.5
2021/12/14 23:51:37 Rebalance progress: 12.5
2021/12/14 23:51:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:51:42 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:51:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:51:57 Rebalance progress: 12.5
2021/12/14 23:52:02 Rebalance progress: 12.5
2021/12/14 23:52:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:52:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 23:52:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:52:13 Rebalance progress: 12.5
2021/12/14 23:52:18 Rebalance progress: 12.5
2021/12/14 23:52:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:52:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:52:28 Rebalance progress: 12.5
2021/12/14 23:52:33 Rebalance progress: 12.5
2021/12/14 23:52:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:52:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:52:43 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 23:52:48 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 23:52:53 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 23:52:58 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 23:52:58 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 23:52:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:52:58 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/14 23:53:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:53:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:53:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:53:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:53:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:53:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8602
2021/12/14 23:53:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.57s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/14 23:53:27 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 23:53:27 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 23:53:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:53:27 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/14 23:53:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:53:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:53:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:53:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:53:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:53:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:53:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:53:45 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:53:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 23:53:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:53:56 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 23:54:01 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 23:54:06 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 23:54:06 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:54:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/14 23:54:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:54:12 Rebalance progress: 12.5
2021/12/14 23:54:17 Rebalance progress: 12.5
2021/12/14 23:54:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:54:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/14 23:54:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:54:27 Rebalance progress: 12.5
2021/12/14 23:54:32 Rebalance progress: 12.5
2021/12/14 23:54:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:54:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 23:54:43 Rebalance progress: 12.5
2021/12/14 23:54:48 Rebalance progress: 12.5
2021/12/14 23:54:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:54:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:54:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 23:54:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 23:55:03 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 23:55:08 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 23:55:13 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 23:55:13 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 23:55:13 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:55:13 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/14 23:55:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:55:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:55:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6117
2021/12/14 23:55:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:55:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:55:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:55:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6117
2021/12/14 23:55:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.80s)
=== RUN   TestEventingRebInWithOps
2021/12/14 23:55:46 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 23:55:46 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/14 23:55:46 eventingSetup.go:46: No eventing-con process running
2021/12/14 23:55:46 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/14 23:55:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:55:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:55:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 23:55:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:56:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:56:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:56:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 23:56:05 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 23:56:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 23:56:11 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 23:56:16 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 23:56:21 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 23:56:26 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 23:56:26 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/14 23:56:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 23:56:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 23:56:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 23:56:32 Rebalance progress: 12.5
2021/12/14 23:56:37 Rebalance progress: 12.5
2021/12/14 23:56:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:56:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:56:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:56:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 23:56:47 Rebalance progress: 12.5
2021/12/14 23:56:52 Rebalance progress: 12.5
2021/12/14 23:56:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:56:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 23:56:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 23:56:57 Waiting for src bucket item count to get to: 200000 curr count: 35347
2021/12/14 23:57:02 Waiting for src bucket item count to get to: 200000 curr count: 50990
2021/12/14 23:57:07 Waiting for src bucket item count to get to: 200000 curr count: 50990
2021/12/14 23:57:12 Waiting for src bucket item count to get to: 200000 curr count: 66964
2021/12/14 23:57:17 Waiting for src bucket item count to get to: 200000 curr count: 66964
2021/12/14 23:57:22 Waiting for src bucket item count to get to: 200000 curr count: 81535
2021/12/14 23:57:27 Waiting for src bucket item count to get to: 200000 curr count: 81535
2021/12/14 23:57:32 Waiting for src bucket item count to get to: 200000 curr count: 97621
2021/12/14 23:57:37 Waiting for src bucket item count to get to: 200000 curr count: 97621
2021/12/14 23:57:42 Waiting for src bucket item count to get to: 200000 curr count: 97621
2021/12/14 23:57:47 Waiting for src bucket item count to get to: 200000 curr count: 113863
2021/12/14 23:57:52 Waiting for src bucket item count to get to: 200000 curr count: 113863
2021/12/14 23:57:57 Waiting for src bucket item count to get to: 200000 curr count: 130073
2021/12/14 23:58:02 Waiting for src bucket item count to get to: 200000 curr count: 130073
2021/12/14 23:58:07 Waiting for src bucket item count to get to: 200000 curr count: 146336
2021/12/14 23:58:12 Waiting for src bucket item count to get to: 200000 curr count: 146336
2021/12/14 23:58:17 Waiting for src bucket item count to get to: 200000 curr count: 162523
2021/12/14 23:58:22 Waiting for src bucket item count to get to: 200000 curr count: 162523
2021/12/14 23:58:27 Waiting for src bucket item count to get to: 200000 curr count: 177330
2021/12/14 23:58:32 Waiting for src bucket item count to get to: 200000 curr count: 177330
2021/12/14 23:58:37 Waiting for src bucket item count to get to: 200000 curr count: 193476
2021/12/14 23:58:42 Waiting for src bucket item count to get to: 200000 curr count: 193476
2021/12/14 23:58:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 23:58:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 23:58:47 Starting bucket ops to source bucket
2021/12/15 00:02:37 Waiting for dst bucket item count to get to: 0 curr count: 4779
2021/12/15 00:02:42 Waiting for dst bucket item count to get to: 0 curr count: 4779
2021/12/15 00:02:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:02:42 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:02:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:02:47 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 00:02:52 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 00:02:57 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 00:03:02 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 00:03:02 App: TestEventingRebInWithOps got un-deployed
2021/12/15 00:03:02 No eventing-con process running
2021/12/15 00:03:02 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/15 00:03:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:03:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:03:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:03:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.34s)
=== RUN   TestEventingRebOutWithOps
2021/12/15 00:03:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 00:03:21 Removing node(s): [] from the cluster
2021/12/15 00:03:26 Rebalance progress: 100
2021/12/15 00:03:31 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 00:03:31 App: TestEventingRebOutWithOps got un-deployed
2021/12/15 00:03:31 No eventing-con process running
2021/12/15 00:03:31 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/15 00:03:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:03:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:03:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:03:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:03:50 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:03:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 00:03:56 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 00:04:01 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 00:04:06 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 00:04:11 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 00:04:11 App: TestEventingRebOutWithOps got deployed
2021/12/15 00:04:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:04:11 Starting bucket ops to source bucket
2021/12/15 00:04:16 Rebalance progress: 12.5
2021/12/15 00:04:21 Rebalance progress: 12.5
2021/12/15 00:04:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:04:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:04:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:04:26 Waiting for src bucket item count to get to: 200000 curr count: 12568
2021/12/15 00:04:31 Waiting for src bucket item count to get to: 200000 curr count: 28150
2021/12/15 00:04:36 Waiting for src bucket item count to get to: 200000 curr count: 28150
2021/12/15 00:04:41 Waiting for src bucket item count to get to: 200000 curr count: 44087
2021/12/15 00:04:46 Waiting for src bucket item count to get to: 200000 curr count: 44087
2021/12/15 00:04:51 Waiting for src bucket item count to get to: 200000 curr count: 60248
2021/12/15 00:04:56 Waiting for src bucket item count to get to: 200000 curr count: 60248
2021/12/15 00:05:01 Waiting for src bucket item count to get to: 200000 curr count: 60248
2021/12/15 00:05:06 Waiting for src bucket item count to get to: 200000 curr count: 75391
2021/12/15 00:05:11 Waiting for src bucket item count to get to: 200000 curr count: 75391
2021/12/15 00:05:16 Waiting for src bucket item count to get to: 200000 curr count: 91017
2021/12/15 00:05:21 Waiting for src bucket item count to get to: 200000 curr count: 91017
2021/12/15 00:05:26 Waiting for src bucket item count to get to: 200000 curr count: 106745
2021/12/15 00:05:31 Waiting for src bucket item count to get to: 200000 curr count: 106745
2021/12/15 00:05:36 Waiting for src bucket item count to get to: 200000 curr count: 122850
2021/12/15 00:05:41 Waiting for src bucket item count to get to: 200000 curr count: 122850
2021/12/15 00:05:46 Waiting for src bucket item count to get to: 200000 curr count: 139198
2021/12/15 00:05:51 Waiting for src bucket item count to get to: 200000 curr count: 139198
2021/12/15 00:05:56 Waiting for src bucket item count to get to: 200000 curr count: 155419
2021/12/15 00:06:01 Waiting for src bucket item count to get to: 200000 curr count: 155419
2021/12/15 00:06:06 Waiting for src bucket item count to get to: 200000 curr count: 170593
2021/12/15 00:06:11 Waiting for src bucket item count to get to: 200000 curr count: 170593
2021/12/15 00:06:16 Waiting for src bucket item count to get to: 200000 curr count: 186425
2021/12/15 00:06:21 Waiting for src bucket item count to get to: 200000 curr count: 186425
2021/12/15 00:06:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 00:06:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 00:06:26 Starting bucket ops to source bucket
2021/12/15 00:10:16 Waiting for dst bucket item count to get to: 0 curr count: 20737
2021/12/15 00:10:21 Waiting for dst bucket item count to get to: 0 curr count: 2783
2021/12/15 00:10:26 Waiting for dst bucket item count to get to: 0 curr count: 2783
2021/12/15 00:10:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:10:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:10:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:10:31 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 00:10:36 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 00:10:41 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 00:10:46 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 00:10:46 App: TestEventingRebOutWithOps got un-deployed
2021/12/15 00:10:46 No eventing-con process running
2021/12/15 00:10:46 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/15 00:10:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:10:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:10:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:10:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:11:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (463.97s)
=== RUN   TestEventingSwapRebWithOps
2021/12/15 00:11:10 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 00:11:10 App: TestEventingSwapRebWithOps got un-deployed
2021/12/15 00:11:10 No eventing-con process running
2021/12/15 00:11:10 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/15 00:11:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:11:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:11:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:11:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:11:28 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:11:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 00:11:34 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 00:11:39 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 00:11:44 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 00:11:49 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 00:11:49 App: TestEventingSwapRebWithOps got deployed
2021/12/15 00:11:49 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 00:11:49 Starting bucket ops to source bucket
2021/12/15 00:11:50 Removing node(s): [] from the cluster
2021/12/15 00:11:55 Rebalance progress: 12.5
2021/12/15 00:12:00 Rebalance progress: 12.5
2021/12/15 00:12:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:12:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 00:12:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:12:11 Rebalance progress: 12.5
2021/12/15 00:12:16 Rebalance progress: 12.5
2021/12/15 00:12:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:12:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 00:12:26 Rebalance progress: 12.5
2021/12/15 00:12:31 Rebalance progress: 12.5
2021/12/15 00:12:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:12:37 Waiting for src bucket item count to get to: 200000 curr count: 60413
2021/12/15 00:12:42 Waiting for src bucket item count to get to: 200000 curr count: 75155
2021/12/15 00:12:47 Waiting for src bucket item count to get to: 200000 curr count: 75155
2021/12/15 00:12:52 Waiting for src bucket item count to get to: 200000 curr count: 90551
2021/12/15 00:12:57 Waiting for src bucket item count to get to: 200000 curr count: 90551
2021/12/15 00:13:02 Waiting for src bucket item count to get to: 200000 curr count: 106823
2021/12/15 00:13:07 Waiting for src bucket item count to get to: 200000 curr count: 106823
2021/12/15 00:13:12 Waiting for src bucket item count to get to: 200000 curr count: 123005
2021/12/15 00:13:17 Waiting for src bucket item count to get to: 200000 curr count: 123005
2021/12/15 00:13:22 Waiting for src bucket item count to get to: 200000 curr count: 139023
2021/12/15 00:13:27 Waiting for src bucket item count to get to: 200000 curr count: 139023
2021/12/15 00:13:32 Waiting for src bucket item count to get to: 200000 curr count: 155348
2021/12/15 00:13:37 Waiting for src bucket item count to get to: 200000 curr count: 155348
2021/12/15 00:13:42 Waiting for src bucket item count to get to: 200000 curr count: 170563
2021/12/15 00:13:47 Waiting for src bucket item count to get to: 200000 curr count: 170563
2021/12/15 00:13:52 Waiting for src bucket item count to get to: 200000 curr count: 186477
2021/12/15 00:13:57 Waiting for src bucket item count to get to: 200000 curr count: 186477
2021/12/15 00:14:02 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 00:14:02 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 00:14:02 Starting bucket ops to source bucket
2021/12/15 00:17:52 Waiting for dst bucket item count to get to: 0 curr count: 13834
2021/12/15 00:17:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:17:52 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:17:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:17:57 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 00:18:02 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 00:18:07 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 00:18:12 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 00:18:12 App: TestEventingSwapRebWithOps got un-deployed
2021/12/15 00:18:12 No eventing-con process running
2021/12/15 00:18:12 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/15 00:18:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:18:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:18:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:18:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:18:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (446.19s)
=== RUN   TestKVRebInWithOps
2021/12/15 00:18:36 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 00:18:36 App: TestKVRebInWithOps got un-deployed
2021/12/15 00:18:36 No eventing-con process running
2021/12/15 00:18:36 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/15 00:18:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:18:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:18:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:18:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:18:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:18:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:18:54 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:18:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 00:19:01 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 00:19:06 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 00:19:11 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 00:19:16 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 00:19:16 App: TestKVRebInWithOps got deployed
2021/12/15 00:19:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 00:19:16 Starting bucket ops to source bucket
2021/12/15 00:19:16 Removing node(s): [] from the cluster
2021/12/15 00:19:21 Rebalance progress: 12.5
2021/12/15 00:19:26 Rebalance progress: 12.5
2021/12/15 00:19:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:19:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:19:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:19:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:19:36 Rebalance progress: 12.5
2021/12/15 00:19:41 Rebalance progress: 12.5
2021/12/15 00:19:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:19:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:19:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:19:46 Waiting for src bucket item count to get to: 200000 curr count: 36316
2021/12/15 00:19:51 Waiting for src bucket item count to get to: 200000 curr count: 52446
2021/12/15 00:19:56 Waiting for src bucket item count to get to: 200000 curr count: 52446
2021/12/15 00:20:01 Waiting for src bucket item count to get to: 200000 curr count: 68681
2021/12/15 00:20:06 Waiting for src bucket item count to get to: 200000 curr count: 68681
2021/12/15 00:20:11 Waiting for src bucket item count to get to: 200000 curr count: 83202
2021/12/15 00:20:16 Waiting for src bucket item count to get to: 200000 curr count: 83202
2021/12/15 00:20:21 Waiting for src bucket item count to get to: 200000 curr count: 83202
2021/12/15 00:20:26 Waiting for src bucket item count to get to: 200000 curr count: 99000
2021/12/15 00:20:31 Waiting for src bucket item count to get to: 200000 curr count: 99000
2021/12/15 00:20:36 Waiting for src bucket item count to get to: 200000 curr count: 114959
2021/12/15 00:20:41 Waiting for src bucket item count to get to: 200000 curr count: 114959
2021/12/15 00:20:46 Waiting for src bucket item count to get to: 200000 curr count: 131196
2021/12/15 00:20:51 Waiting for src bucket item count to get to: 200000 curr count: 131196
2021/12/15 00:20:56 Waiting for src bucket item count to get to: 200000 curr count: 147441
2021/12/15 00:21:01 Waiting for src bucket item count to get to: 200000 curr count: 147441
2021/12/15 00:21:06 Waiting for src bucket item count to get to: 200000 curr count: 163563
2021/12/15 00:21:11 Waiting for src bucket item count to get to: 200000 curr count: 163563
2021/12/15 00:21:16 Waiting for src bucket item count to get to: 200000 curr count: 178569
2021/12/15 00:21:21 Waiting for src bucket item count to get to: 200000 curr count: 178569
2021/12/15 00:21:26 Waiting for src bucket item count to get to: 200000 curr count: 194480
2021/12/15 00:21:31 Waiting for src bucket item count to get to: 200000 curr count: 194480
2021/12/15 00:21:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 00:21:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 00:21:36 Starting bucket ops to source bucket
2021/12/15 00:25:27 Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/15 00:25:32 Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/15 00:25:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:25:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:25:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:25:37 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 00:25:42 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 00:25:47 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 00:25:52 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 00:25:52 App: TestKVRebInWithOps got un-deployed
2021/12/15 00:25:52 No eventing-con process running
2021/12/15 00:25:52 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/15 00:25:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:25:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:25:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:26:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:26:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:26:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.67s)
=== RUN   TestKVRebOutWithOps
2021/12/15 00:26:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 00:26:11 Removing node(s): [] from the cluster
2021/12/15 00:26:16 Rebalance progress: 100
2021/12/15 00:26:21 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 00:26:21 App: TestKVRebOutWithOps got un-deployed
2021/12/15 00:26:21 No eventing-con process running
2021/12/15 00:26:21 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/15 00:26:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:26:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:26:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:26:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:26:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:26:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:26:39 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:26:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 00:26:45 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 00:26:50 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 00:26:55 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 00:27:00 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 00:27:00 App: TestKVRebOutWithOps got deployed
2021/12/15 00:27:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:27:01 Starting bucket ops to source bucket
2021/12/15 00:27:06 Rebalance progress: 12.5
2021/12/15 00:27:11 Rebalance progress: 12.5
2021/12/15 00:27:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:27:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:27:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:27:16 Waiting for src bucket item count to get to: 200000 curr count: 12701
2021/12/15 00:27:21 Waiting for src bucket item count to get to: 200000 curr count: 28314
2021/12/15 00:27:26 Waiting for src bucket item count to get to: 200000 curr count: 28314
2021/12/15 00:27:31 Waiting for src bucket item count to get to: 200000 curr count: 44279
2021/12/15 00:27:36 Waiting for src bucket item count to get to: 200000 curr count: 44279
2021/12/15 00:27:41 Waiting for src bucket item count to get to: 200000 curr count: 44279
2021/12/15 00:27:46 Waiting for src bucket item count to get to: 200000 curr count: 60324
2021/12/15 00:27:51 Waiting for src bucket item count to get to: 200000 curr count: 60324
2021/12/15 00:27:56 Waiting for src bucket item count to get to: 200000 curr count: 75575
2021/12/15 00:28:01 Waiting for src bucket item count to get to: 200000 curr count: 75575
2021/12/15 00:28:06 Waiting for src bucket item count to get to: 200000 curr count: 91401
2021/12/15 00:28:11 Waiting for src bucket item count to get to: 200000 curr count: 91401
2021/12/15 00:28:16 Waiting for src bucket item count to get to: 200000 curr count: 107607
2021/12/15 00:28:21 Waiting for src bucket item count to get to: 200000 curr count: 107607
2021/12/15 00:28:26 Waiting for src bucket item count to get to: 200000 curr count: 123587
2021/12/15 00:28:31 Waiting for src bucket item count to get to: 200000 curr count: 123587
2021/12/15 00:28:36 Waiting for src bucket item count to get to: 200000 curr count: 139819
2021/12/15 00:28:41 Waiting for src bucket item count to get to: 200000 curr count: 139819
2021/12/15 00:28:46 Waiting for src bucket item count to get to: 200000 curr count: 156069
2021/12/15 00:28:51 Waiting for src bucket item count to get to: 200000 curr count: 156069
2021/12/15 00:28:56 Waiting for src bucket item count to get to: 200000 curr count: 171279
2021/12/15 00:29:01 Waiting for src bucket item count to get to: 200000 curr count: 171279
2021/12/15 00:29:06 Waiting for src bucket item count to get to: 200000 curr count: 187177
2021/12/15 00:29:11 Waiting for src bucket item count to get to: 200000 curr count: 187177
2021/12/15 00:29:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 00:29:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 00:29:16 Starting bucket ops to source bucket
2021/12/15 00:33:06 Waiting for dst bucket item count to get to: 0 curr count: 2722
2021/12/15 00:33:11 Waiting for dst bucket item count to get to: 0 curr count: 2722
2021/12/15 00:33:16 Waiting for dst bucket item count to get to: 0 curr count: 2722
2021/12/15 00:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:33:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:33:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:33:21 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 00:33:26 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 00:33:31 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 00:33:36 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 00:33:36 App: TestKVRebOutWithOps got un-deployed
2021/12/15 00:33:36 No eventing-con process running
2021/12/15 00:33:36 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/15 00:33:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:33:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:33:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:33:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:33:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:33:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (464.10s)
=== RUN   TestKVSwapRebWithOps
2021/12/15 00:34:00 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 00:34:00 App: TestKVSwapRebWithOps got un-deployed
2021/12/15 00:34:00 No eventing-con process running
2021/12/15 00:34:00 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/15 00:34:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:34:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:34:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:34:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:34:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:34:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:34:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:34:18 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:34:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 00:34:24 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 00:34:29 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 00:34:34 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 00:34:39 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 00:34:39 App: TestKVSwapRebWithOps got deployed
2021/12/15 00:34:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 00:34:39 Starting bucket ops to source bucket
2021/12/15 00:34:40 Removing node(s): [] from the cluster
2021/12/15 00:34:45 Rebalance progress: 12.5
2021/12/15 00:34:50 Rebalance progress: 12.5
2021/12/15 00:34:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:34:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:34:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:34:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 00:34:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:35:00 Rebalance progress: 12.5
2021/12/15 00:35:05 Rebalance progress: 12.5
2021/12/15 00:35:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:35:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:35:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 00:35:15 Rebalance progress: 12.5
2021/12/15 00:35:20 Rebalance progress: 12.5
2021/12/15 00:35:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:35:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:35:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:35:25 Waiting for src bucket item count to get to: 200000 curr count: 61151
2021/12/15 00:35:30 Waiting for src bucket item count to get to: 200000 curr count: 76074
2021/12/15 00:35:35 Waiting for src bucket item count to get to: 200000 curr count: 76074
2021/12/15 00:35:40 Waiting for src bucket item count to get to: 200000 curr count: 91841
2021/12/15 00:35:45 Waiting for src bucket item count to get to: 200000 curr count: 91841
2021/12/15 00:35:50 Waiting for src bucket item count to get to: 200000 curr count: 107975
2021/12/15 00:35:55 Waiting for src bucket item count to get to: 200000 curr count: 107975
2021/12/15 00:36:00 Waiting for src bucket item count to get to: 200000 curr count: 124371
2021/12/15 00:36:05 Waiting for src bucket item count to get to: 200000 curr count: 124371
2021/12/15 00:36:10 Waiting for src bucket item count to get to: 200000 curr count: 140432
2021/12/15 00:36:15 Waiting for src bucket item count to get to: 200000 curr count: 140432
2021/12/15 00:36:20 Waiting for src bucket item count to get to: 200000 curr count: 140432
2021/12/15 00:36:25 Waiting for src bucket item count to get to: 200000 curr count: 156447
2021/12/15 00:36:30 Waiting for src bucket item count to get to: 200000 curr count: 156447
2021/12/15 00:36:35 Waiting for src bucket item count to get to: 200000 curr count: 171175
2021/12/15 00:36:40 Waiting for src bucket item count to get to: 200000 curr count: 171175
2021/12/15 00:36:45 Waiting for src bucket item count to get to: 200000 curr count: 186851
2021/12/15 00:36:50 Waiting for src bucket item count to get to: 200000 curr count: 186851
2021/12/15 00:36:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 00:36:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 00:36:55 Starting bucket ops to source bucket
2021/12/15 00:40:45 Waiting for dst bucket item count to get to: 0 curr count: 2056
2021/12/15 00:40:50 Waiting for dst bucket item count to get to: 0 curr count: 2056
2021/12/15 00:40:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:40:50 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:40:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:40:56 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 00:41:01 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 00:41:06 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 00:41:11 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 00:41:11 App: TestKVSwapRebWithOps got un-deployed
2021/12/15 00:41:11 No eventing-con process running
2021/12/15 00:41:11 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/15 00:41:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:41:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:41:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:41:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:41:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (454.62s)
=== RUN   TestEventingRebInWithTimers
2021/12/15 00:41:34 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 00:41:34 App: TestEventingRebInWithTimers got un-deployed
2021/12/15 00:41:34 No eventing-con process running
2021/12/15 00:41:34 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/15 00:41:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:41:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:41:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:41:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:41:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:41:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:41:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:41:53 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:41:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 00:41:59 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 00:42:04 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 00:42:09 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 00:42:14 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 00:42:14 App: TestEventingRebInWithTimers got deployed
2021/12/15 00:42:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 00:42:14 Starting bucket ops to source bucket
2021/12/15 00:42:16 Removing node(s): [] from the cluster
2021/12/15 00:42:21 Rebalance progress: 12.5
2021/12/15 00:42:26 Rebalance progress: 12.5
2021/12/15 00:42:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:42:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:42:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:42:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:42:36 Rebalance progress: 12.5
2021/12/15 00:42:41 Rebalance progress: 12.5
2021/12/15 00:42:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:42:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:42:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:42:46 Waiting for src bucket item count to get to: 50000 curr count: 30701
2021/12/15 00:42:51 Waiting for src bucket item count to get to: 50000 curr count: 30701
2021/12/15 00:42:56 Waiting for src bucket item count to get to: 50000 curr count: 42001
2021/12/15 00:43:01 Waiting for src bucket item count to get to: 50000 curr count: 42001
2021/12/15 00:43:11 Waiting for dst bucket item count to get to: 50000 curr count: 37086
2021/12/15 00:43:16 Waiting for dst bucket item count to get to: 50000 curr count: 37086
2021/12/15 00:43:21 Waiting for dst bucket item count to get to: 50000 curr count: 45755
2021/12/15 00:43:26 Waiting for dst bucket item count to get to: 50000 curr count: 45755
2021/12/15 00:43:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 00:43:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 00:43:26 Starting bucket ops to source bucket
2021/12/15 00:44:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:44:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:44:46 Waiting for dst bucket item count to get to: 0 curr count: 49835
2021/12/15 00:44:51 Waiting for dst bucket item count to get to: 0 curr count: 49835
2021/12/15 00:44:56 Waiting for dst bucket item count to get to: 0 curr count: 49835
2021/12/15 00:45:01 Waiting for dst bucket item count to get to: 0 curr count: 31764
2021/12/15 00:45:06 Waiting for dst bucket item count to get to: 0 curr count: 31764
2021/12/15 00:45:11 Waiting for dst bucket item count to get to: 0 curr count: 16022
2021/12/15 00:45:16 Waiting for dst bucket item count to get to: 0 curr count: 16022
2021/12/15 00:45:21 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/15 00:45:26 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/15 00:45:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:45:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:45:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:45:31 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 00:45:36 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 00:45:41 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 00:45:46 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 00:45:46 App: TestEventingRebInWithTimers got un-deployed
2021/12/15 00:45:46 No eventing-con process running
2021/12/15 00:45:46 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/15 00:45:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:45:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:45:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:45:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:46:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:46:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (275.61s)
=== RUN   TestEventingRebOutWithTimers
2021/12/15 00:46:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 00:46:05 Removing node(s): [] from the cluster
2021/12/15 00:46:10 Rebalance progress: 100
2021/12/15 00:46:15 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 00:46:15 App: TestEventingRebOutWithTimers got un-deployed
2021/12/15 00:46:15 No eventing-con process running
2021/12/15 00:46:15 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/15 00:46:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:46:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:46:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:46:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:46:34 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:46:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 00:46:40 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 00:46:45 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 00:46:50 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 00:46:55 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 00:46:55 App: TestEventingRebOutWithTimers got deployed
2021/12/15 00:46:55 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:46:55 Starting bucket ops to source bucket
2021/12/15 00:47:00 Rebalance progress: 12.5
2021/12/15 00:47:05 Rebalance progress: 12.5
2021/12/15 00:47:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:47:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:47:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:47:10 Waiting for src bucket item count to get to: 50000 curr count: 18996
2021/12/15 00:47:15 Waiting for src bucket item count to get to: 50000 curr count: 18996
2021/12/15 00:47:20 Waiting for src bucket item count to get to: 50000 curr count: 30359
2021/12/15 00:47:25 Waiting for src bucket item count to get to: 50000 curr count: 30359
2021/12/15 00:47:30 Waiting for src bucket item count to get to: 50000 curr count: 42121
2021/12/15 00:47:35 Waiting for src bucket item count to get to: 50000 curr count: 42121
2021/12/15 00:47:40 Waiting for src bucket item count to get to: 50000 curr count: 42121
2021/12/15 00:47:50 Waiting for dst bucket item count to get to: 50000 curr count: 35459
2021/12/15 00:47:55 Waiting for dst bucket item count to get to: 50000 curr count: 35459
2021/12/15 00:48:00 Waiting for dst bucket item count to get to: 50000 curr count: 44998
2021/12/15 00:48:05 Waiting for dst bucket item count to get to: 50000 curr count: 44998
2021/12/15 00:48:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 00:48:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 00:48:05 Starting bucket ops to source bucket
2021/12/15 00:49:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:49:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:49:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:49:31 Waiting for dst bucket item count to get to: 0 curr count: 48364
2021/12/15 00:49:36 Waiting for dst bucket item count to get to: 0 curr count: 48364
2021/12/15 00:49:41 Waiting for dst bucket item count to get to: 0 curr count: 34401
2021/12/15 00:49:46 Waiting for dst bucket item count to get to: 0 curr count: 34401
2021/12/15 00:49:51 Waiting for dst bucket item count to get to: 0 curr count: 15796
2021/12/15 00:49:56 Waiting for dst bucket item count to get to: 0 curr count: 15796
2021/12/15 00:50:01 Waiting for dst bucket item count to get to: 0 curr count: 3755
2021/12/15 00:50:06 Waiting for dst bucket item count to get to: 0 curr count: 3755
2021/12/15 00:50:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:50:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:50:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:50:11 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 00:50:16 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 00:50:21 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 00:50:26 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 00:50:26 App: TestEventingRebOutWithTimers got un-deployed
2021/12/15 00:50:26 No eventing-con process running
2021/12/15 00:50:26 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/15 00:50:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:50:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:50:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:50:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:50:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:50:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.83s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/15 00:50:50 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 00:50:50 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/15 00:50:50 No eventing-con process running
2021/12/15 00:50:50 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/15 00:50:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:50:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:50:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:51:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:51:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:51:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:51:08 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:51:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 00:51:14 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 00:51:19 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 00:51:24 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 00:51:29 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 00:51:29 App: TestEventingSwapRebWithTimers got deployed
2021/12/15 00:51:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 00:51:29 Starting bucket ops to source bucket
2021/12/15 00:51:30 Removing node(s): [] from the cluster
2021/12/15 00:51:35 Rebalance progress: 12.5
2021/12/15 00:51:40 Rebalance progress: 12.5
2021/12/15 00:51:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:51:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:51:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:51:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 00:51:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:51:51 Rebalance progress: 12.5
2021/12/15 00:51:56 Rebalance progress: 12.5
2021/12/15 00:52:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:52:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 00:52:06 Rebalance progress: 12.5
2021/12/15 00:52:11 Rebalance progress: 12.5
2021/12/15 00:52:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:52:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:52:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:52:16 Waiting for src bucket item count to get to: 50000 curr count: 47017
2021/12/15 00:52:26 Waiting for dst bucket item count to get to: 50000 curr count: 40741
2021/12/15 00:52:31 Waiting for dst bucket item count to get to: 50000 curr count: 40741
2021/12/15 00:52:36 Waiting for dst bucket item count to get to: 50000 curr count: 49098
2021/12/15 00:52:41 Waiting for dst bucket item count to get to: 50000 curr count: 49098
2021/12/15 00:52:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 00:52:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 00:52:41 Starting bucket ops to source bucket
2021/12/15 00:53:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:53:56 Waiting for dst bucket item count to get to: 0 curr count: 49722
2021/12/15 00:54:01 Waiting for dst bucket item count to get to: 0 curr count: 49722
2021/12/15 00:54:06 Waiting for dst bucket item count to get to: 0 curr count: 49722
2021/12/15 00:54:11 Waiting for dst bucket item count to get to: 0 curr count: 40393
2021/12/15 00:54:16 Waiting for dst bucket item count to get to: 0 curr count: 40393
2021/12/15 00:54:21 Waiting for dst bucket item count to get to: 0 curr count: 30376
2021/12/15 00:54:26 Waiting for dst bucket item count to get to: 0 curr count: 30376
2021/12/15 00:54:31 Waiting for dst bucket item count to get to: 0 curr count: 9460
2021/12/15 00:54:36 Waiting for dst bucket item count to get to: 0 curr count: 9460
2021/12/15 00:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:54:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:54:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:54:41 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 00:54:46 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 00:54:52 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 00:54:57 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 00:54:57 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/15 00:54:57 No eventing-con process running
2021/12/15 00:54:57 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/15 00:54:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:55:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:55:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:55:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:55:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:55:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (270.50s)
=== RUN   TestKVRebInWithTimers
2021/12/15 00:55:20 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 00:55:20 App: TestKVRebInWithTimers got un-deployed
2021/12/15 00:55:20 No eventing-con process running
2021/12/15 00:55:20 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/15 00:55:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:55:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:55:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:55:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:55:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:55:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:55:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:55:39 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 00:55:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 00:55:45 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 00:55:50 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 00:55:55 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 00:56:00 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 00:56:00 App: TestKVRebInWithTimers got deployed
2021/12/15 00:56:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 00:56:00 Starting bucket ops to source bucket
2021/12/15 00:56:00 Removing node(s): [] from the cluster
2021/12/15 00:56:05 Rebalance progress: 12.5
2021/12/15 00:56:10 Rebalance progress: 12.5
2021/12/15 00:56:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:56:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:56:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:56:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 00:56:20 Rebalance progress: 12.5
2021/12/15 00:56:25 Rebalance progress: 12.5
2021/12/15 00:56:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 00:56:30 Waiting for src bucket item count to get to: 50000 curr count: 34478
2021/12/15 00:56:35 Waiting for src bucket item count to get to: 50000 curr count: 34478
2021/12/15 00:56:40 Waiting for src bucket item count to get to: 50000 curr count: 46159
2021/12/15 00:56:45 Waiting for src bucket item count to get to: 50000 curr count: 46159
2021/12/15 00:56:55 Waiting for dst bucket item count to get to: 50000 curr count: 41139
2021/12/15 00:57:00 Waiting for dst bucket item count to get to: 50000 curr count: 41139
2021/12/15 00:57:05 Waiting for dst bucket item count to get to: 50000 curr count: 41139
2021/12/15 00:57:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 00:57:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 00:57:05 Starting bucket ops to source bucket
2021/12/15 00:58:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:58:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:58:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 00:58:31 Waiting for dst bucket item count to get to: 0 curr count: 49454
2021/12/15 00:58:36 Waiting for dst bucket item count to get to: 0 curr count: 49454
2021/12/15 00:58:41 Waiting for dst bucket item count to get to: 0 curr count: 31813
2021/12/15 00:58:46 Waiting for dst bucket item count to get to: 0 curr count: 31813
2021/12/15 00:58:51 Waiting for dst bucket item count to get to: 0 curr count: 17985
2021/12/15 00:58:56 Waiting for dst bucket item count to get to: 0 curr count: 17985
2021/12/15 00:58:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:58:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 00:58:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 00:59:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 00:59:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 00:59:11 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 00:59:16 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 00:59:16 App: TestKVRebInWithTimers got un-deployed
2021/12/15 00:59:16 No eventing-con process running
2021/12/15 00:59:17 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/15 00:59:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:59:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:59:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:59:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:59:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:59:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.92s)
=== RUN   TestKVRebOutWithTimers
2021/12/15 00:59:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 00:59:36 Removing node(s): [] from the cluster
2021/12/15 00:59:41 Rebalance progress: 100
2021/12/15 00:59:46 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 00:59:46 App: TestKVRebOutWithTimers got un-deployed
2021/12/15 00:59:46 No eventing-con process running
2021/12/15 00:59:46 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/15 00:59:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:59:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:59:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 00:59:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 00:59:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 00:59:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:00:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:00:04 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:00:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 01:00:10 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 01:00:15 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 01:00:20 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 01:00:25 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 01:00:25 App: TestKVRebOutWithTimers got deployed
2021/12/15 01:00:25 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:00:25 Starting bucket ops to source bucket
2021/12/15 01:00:30 Rebalance progress: 12.5
2021/12/15 01:00:35 Rebalance progress: 12.5
2021/12/15 01:00:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:00:40 Waiting for src bucket item count to get to: 50000 curr count: 19063
2021/12/15 01:00:45 Waiting for src bucket item count to get to: 50000 curr count: 19063
2021/12/15 01:00:50 Waiting for src bucket item count to get to: 50000 curr count: 29651
2021/12/15 01:00:55 Waiting for src bucket item count to get to: 50000 curr count: 29651
2021/12/15 01:01:00 Waiting for src bucket item count to get to: 50000 curr count: 41703
2021/12/15 01:01:05 Waiting for src bucket item count to get to: 50000 curr count: 41703
2021/12/15 01:01:10 Waiting for src bucket item count to get to: 50000 curr count: 41703
2021/12/15 01:01:21 Waiting for dst bucket item count to get to: 50000 curr count: 38959
2021/12/15 01:01:26 Waiting for dst bucket item count to get to: 50000 curr count: 38959
2021/12/15 01:01:31 Waiting for dst bucket item count to get to: 50000 curr count: 47127
2021/12/15 01:01:36 Waiting for dst bucket item count to get to: 50000 curr count: 47127
2021/12/15 01:01:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:01:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:01:36 Starting bucket ops to source bucket
2021/12/15 01:02:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:02:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:02:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:03:01 Waiting for dst bucket item count to get to: 0 curr count: 48219
2021/12/15 01:03:06 Waiting for dst bucket item count to get to: 0 curr count: 48219
2021/12/15 01:03:11 Waiting for dst bucket item count to get to: 0 curr count: 36202
2021/12/15 01:03:16 Waiting for dst bucket item count to get to: 0 curr count: 36202
2021/12/15 01:03:21 Waiting for dst bucket item count to get to: 0 curr count: 12707
2021/12/15 01:03:26 Waiting for dst bucket item count to get to: 0 curr count: 12707
2021/12/15 01:03:31 Waiting for dst bucket item count to get to: 0 curr count: 900
2021/12/15 01:03:36 Waiting for dst bucket item count to get to: 0 curr count: 900
2021/12/15 01:03:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:03:36 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 01:03:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:03:41 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 01:03:46 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 01:03:51 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 01:03:56 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 01:03:56 App: TestKVRebOutWithTimers got un-deployed
2021/12/15 01:03:56 No eventing-con process running
2021/12/15 01:03:56 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/15 01:03:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:04:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:04:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:04:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:04:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:04:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.79s)
=== RUN   TestKVSwapRebWithTimers
2021/12/15 01:04:20 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:04:20 App: TestKVSwapRebWithTimers got un-deployed
2021/12/15 01:04:20 No eventing-con process running
2021/12/15 01:04:20 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/15 01:04:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:04:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:04:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:04:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:04:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:04:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:04:38 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:04:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 01:04:45 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 01:04:50 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 01:04:55 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 01:05:00 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 01:05:00 App: TestKVSwapRebWithTimers got deployed
2021/12/15 01:05:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:05:00 Starting bucket ops to source bucket
2021/12/15 01:05:00 Removing node(s): [] from the cluster
2021/12/15 01:05:05 Rebalance progress: 12.5
2021/12/15 01:05:10 Rebalance progress: 12.5
2021/12/15 01:05:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:05:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:05:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:05:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 01:05:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:05:21 Rebalance progress: 12.5
2021/12/15 01:05:26 Rebalance progress: 12.5
2021/12/15 01:05:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:05:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:05:36 Rebalance progress: 12.5
2021/12/15 01:05:41 Rebalance progress: 12.5
2021/12/15 01:05:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:05:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:05:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:05:46 Waiting for src bucket item count to get to: 50000 curr count: 45839
2021/12/15 01:05:56 Waiting for dst bucket item count to get to: 50000 curr count: 41533
2021/12/15 01:06:01 Waiting for dst bucket item count to get to: 50000 curr count: 41533
2021/12/15 01:06:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:06:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:06:01 Starting bucket ops to source bucket
2021/12/15 01:07:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:07:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:07:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:07:27 Waiting for dst bucket item count to get to: 0 curr count: 43898
2021/12/15 01:07:32 Waiting for dst bucket item count to get to: 0 curr count: 43898
2021/12/15 01:07:37 Waiting for dst bucket item count to get to: 0 curr count: 25263
2021/12/15 01:07:42 Waiting for dst bucket item count to get to: 0 curr count: 25263
2021/12/15 01:07:47 Waiting for dst bucket item count to get to: 0 curr count: 25263
2021/12/15 01:07:52 Waiting for dst bucket item count to get to: 0 curr count: 11751
2021/12/15 01:07:57 Waiting for dst bucket item count to get to: 0 curr count: 11751
2021/12/15 01:07:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:07:57 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 01:07:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:08:02 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:08:07 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:08:12 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:08:17 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:08:22 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:08:27 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 01:08:27 App: TestKVSwapRebWithTimers got un-deployed
2021/12/15 01:08:27 No eventing-con process running
2021/12/15 01:08:27 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/15 01:08:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:08:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:08:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:08:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:08:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (270.53s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/15 01:08:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:08:50 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/15 01:08:50 No eventing-con process running
2021/12/15 01:08:50 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/15 01:08:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:08:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:08:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:09:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:09:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:09:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:09:09 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:09:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 01:09:15 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 01:09:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 01:09:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 01:09:30 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 01:09:30 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/15 01:09:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 01:09:30 Starting bucket ops to source bucket
2021/12/15 01:09:30 Removing node(s): [] from the cluster
2021/12/15 01:09:35 Rebalance progress: 12.5
2021/12/15 01:09:40 Rebalance progress: 12.5
2021/12/15 01:09:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:09:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:09:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:09:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:09:51 Rebalance progress: 12.5
2021/12/15 01:09:56 Rebalance progress: 12.5
2021/12/15 01:10:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:10:01 Waiting for src bucket item count to get to: 50000 curr count: 35801
2021/12/15 01:10:06 Waiting for src bucket item count to get to: 50000 curr count: 35801
2021/12/15 01:10:11 Waiting for src bucket item count to get to: 50000 curr count: 45191
2021/12/15 01:10:16 Waiting for src bucket item count to get to: 50000 curr count: 45191
2021/12/15 01:10:26 Waiting for dst bucket item count to get to: 50000 curr count: 35028
2021/12/15 01:10:31 Waiting for dst bucket item count to get to: 50000 curr count: 35028
2021/12/15 01:10:36 Waiting for dst bucket item count to get to: 50000 curr count: 45421
2021/12/15 01:10:41 Waiting for dst bucket item count to get to: 50000 curr count: 45421
2021/12/15 01:10:46 Waiting for dst bucket item count to get to: 50000 curr count: 45421
2021/12/15 01:10:46 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:10:46 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:10:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:10:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:10:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:11:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:11:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:11:11 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:11:16 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 01:11:16 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/15 01:11:16 No eventing-con process running
2021/12/15 01:11:16 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/15 01:11:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:11:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:11:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:11:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:11:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (174.13s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/15 01:11:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 01:11:40 Removing node(s): [] from the cluster
2021/12/15 01:11:45 Rebalance progress: 100
2021/12/15 01:11:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:11:50 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/15 01:11:50 No eventing-con process running
2021/12/15 01:11:50 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/15 01:11:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:11:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:11:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:12:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:12:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:12:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:12:08 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:12:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 01:12:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 01:12:20 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 01:12:25 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 01:12:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 01:12:30 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/15 01:12:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:12:30 Starting bucket ops to source bucket
2021/12/15 01:12:35 Rebalance progress: 12.5
2021/12/15 01:12:40 Rebalance progress: 12.5
2021/12/15 01:12:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:12:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:12:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:12:45 Waiting for src bucket item count to get to: 50000 curr count: 12881
2021/12/15 01:12:50 Waiting for src bucket item count to get to: 50000 curr count: 12881
2021/12/15 01:12:55 Waiting for src bucket item count to get to: 50000 curr count: 26859
2021/12/15 01:13:00 Waiting for src bucket item count to get to: 50000 curr count: 38819
2021/12/15 01:13:05 Waiting for src bucket item count to get to: 50000 curr count: 38819
2021/12/15 01:13:10 Waiting for src bucket item count to get to: 50000 curr count: 38819
2021/12/15 01:13:15 Waiting for src bucket item count to get to: 50000 curr count: 49060
2021/12/15 01:13:20 Waiting for src bucket item count to get to: 50000 curr count: 49060
2021/12/15 01:13:30 Waiting for dst bucket item count to get to: 50000 curr count: 34125
2021/12/15 01:13:35 Waiting for dst bucket item count to get to: 50000 curr count: 34125
2021/12/15 01:13:40 Waiting for dst bucket item count to get to: 50000 curr count: 44481
2021/12/15 01:13:45 Waiting for dst bucket item count to get to: 50000 curr count: 44481
2021/12/15 01:13:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:13:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:13:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:13:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:13:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:14:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:14:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:14:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:14:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 01:14:15 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/15 01:14:15 No eventing-con process running
2021/12/15 01:14:15 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/15 01:14:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:14:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:14:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:14:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:14:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:14:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:14:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:14:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.27s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/15 01:14:49 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:14:49 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/15 01:14:49 No eventing-con process running
2021/12/15 01:14:49 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/15 01:14:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:14:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:14:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:15:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:15:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:15:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:15:07 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:15:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 01:15:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 01:15:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 01:15:24 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 01:15:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 01:15:29 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/15 01:15:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 01:15:29 Starting bucket ops to source bucket
2021/12/15 01:15:29 Removing node(s): [] from the cluster
2021/12/15 01:15:34 Rebalance progress: 12.5
2021/12/15 01:15:39 Rebalance progress: 12.5
2021/12/15 01:15:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:15:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 01:15:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:15:50 Rebalance progress: 12.5
2021/12/15 01:15:55 Rebalance progress: 12.5
2021/12/15 01:16:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:16:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:16:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:16:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:16:05 Rebalance progress: 12.5
2021/12/15 01:16:10 Rebalance progress: 12.5
2021/12/15 01:16:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:16:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:16:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:16:15 Waiting for src bucket item count to get to: 50000 curr count: 47455
2021/12/15 01:16:20 Waiting for src bucket item count to get to: 50000 curr count: 47455
2021/12/15 01:16:30 Waiting for dst bucket item count to get to: 50000 curr count: 36418
2021/12/15 01:16:35 Waiting for dst bucket item count to get to: 50000 curr count: 36418
2021/12/15 01:16:40 Waiting for dst bucket item count to get to: 50000 curr count: 43367
2021/12/15 01:16:45 Waiting for dst bucket item count to get to: 50000 curr count: 43367
2021/12/15 01:16:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:16:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:16:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:16:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:16:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:17:00 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:17:05 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:17:10 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:17:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 01:17:15 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/15 01:17:15 No eventing-con process running
2021/12/15 01:17:15 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/15 01:17:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:17:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:17:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:17:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:17:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:17:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.92s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/15 01:17:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:17:49 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/15 01:17:49 No eventing-con process running
2021/12/15 01:17:49 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/15 01:17:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:17:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:17:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:18:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:18:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:18:07 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:18:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 01:18:13 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 01:18:19 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 01:18:24 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 01:18:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 01:18:29 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/15 01:18:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:18:29 Starting bucket ops to source bucket
2021/12/15 01:18:29 Removing node(s): [] from the cluster
2021/12/15 01:18:34 Rebalance progress: 12.5
2021/12/15 01:18:39 Rebalance progress: 12.5
2021/12/15 01:18:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:18:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:18:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:18:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:18:49 Rebalance progress: 12.5
2021/12/15 01:18:54 Rebalance progress: 12.5
2021/12/15 01:18:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:18:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:18:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:18:59 Waiting for src bucket item count to get to: 50000 curr count: 26509
2021/12/15 01:19:04 Waiting for src bucket item count to get to: 50000 curr count: 36667
2021/12/15 01:19:09 Waiting for src bucket item count to get to: 50000 curr count: 36667
2021/12/15 01:19:14 Waiting for src bucket item count to get to: 50000 curr count: 47911
2021/12/15 01:19:19 Waiting for src bucket item count to get to: 50000 curr count: 47911
2021/12/15 01:19:29 Waiting for dst bucket item count to get to: 50000 curr count: 39292
2021/12/15 01:19:34 Waiting for dst bucket item count to get to: 50000 curr count: 39292
2021/12/15 01:19:39 Waiting for dst bucket item count to get to: 50000 curr count: 46796
2021/12/15 01:19:44 Waiting for dst bucket item count to get to: 50000 curr count: 46796
2021/12/15 01:19:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:19:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:19:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:19:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:19:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:20:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:20:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:20:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:20:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 01:20:15 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/15 01:20:15 No eventing-con process running
2021/12/15 01:20:15 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/15 01:20:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:20:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:20:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:20:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:20:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.46s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/15 01:20:43 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:20:44 Removing node(s): [] from the cluster
2021/12/15 01:20:49 Rebalance progress: 100
2021/12/15 01:20:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:20:54 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/15 01:20:54 No eventing-con process running
2021/12/15 01:20:54 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/15 01:20:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:21:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:21:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:21:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:21:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:21:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:21:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:21:12 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:21:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 01:21:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 01:21:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 01:21:28 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 01:21:33 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 01:21:34 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/15 01:21:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:21:34 Starting bucket ops to source bucket
2021/12/15 01:21:39 Rebalance progress: 12.5
2021/12/15 01:21:44 Rebalance progress: 12.5
2021/12/15 01:21:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:21:49 Waiting for src bucket item count to get to: 50000 curr count: 7034
2021/12/15 01:21:54 Waiting for src bucket item count to get to: 50000 curr count: 21022
2021/12/15 01:21:59 Waiting for src bucket item count to get to: 50000 curr count: 21022
2021/12/15 01:22:04 Waiting for src bucket item count to get to: 50000 curr count: 34339
2021/12/15 01:22:09 Waiting for src bucket item count to get to: 50000 curr count: 34339
2021/12/15 01:22:14 Waiting for src bucket item count to get to: 50000 curr count: 44605
2021/12/15 01:22:19 Waiting for src bucket item count to get to: 50000 curr count: 44605
2021/12/15 01:22:29 Waiting for dst bucket item count to get to: 50000 curr count: 30336
2021/12/15 01:22:34 Waiting for dst bucket item count to get to: 50000 curr count: 30336
2021/12/15 01:22:39 Waiting for dst bucket item count to get to: 50000 curr count: 30336
2021/12/15 01:22:44 Waiting for dst bucket item count to get to: 50000 curr count: 38438
2021/12/15 01:22:49 Waiting for dst bucket item count to get to: 50000 curr count: 38438
2021/12/15 01:22:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:22:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:22:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:22:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:22:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:23:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:23:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:23:14 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:23:19 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 01:23:19 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/15 01:23:19 No eventing-con process running
2021/12/15 01:23:19 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/15 01:23:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:23:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:23:31 Waiting for dst bucket item count to get to: 0 curr count: 43554
2021/12/15 01:23:36 Waiting for dst bucket item count to get to: 0 curr count: 43554
2021/12/15 01:23:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:23:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:23:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:23:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.56s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/15 01:23:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:23:58 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/15 01:23:58 No eventing-con process running
2021/12/15 01:23:58 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/15 01:24:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:24:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:24:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:24:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:24:16 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:24:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 01:24:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 01:24:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 01:24:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 01:24:38 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 01:24:38 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/15 01:24:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:24:38 Starting bucket ops to source bucket
2021/12/15 01:24:39 Removing node(s): [] from the cluster
2021/12/15 01:24:44 Rebalance progress: 12.5
2021/12/15 01:24:49 Rebalance progress: 12.5
2021/12/15 01:24:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:24:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:24:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:24:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 01:24:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:25:00 Rebalance progress: 12.5
2021/12/15 01:25:05 Rebalance progress: 12.5
2021/12/15 01:25:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:25:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:25:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:25:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:25:15 Rebalance progress: 12.5
2021/12/15 01:25:20 Rebalance progress: 12.5
2021/12/15 01:25:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:25:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:25:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:25:25 Waiting for src bucket item count to get to: 50000 curr count: 49792
2021/12/15 01:25:30 Waiting for src bucket item count to get to: 50000 curr count: 49792
2021/12/15 01:25:40 Waiting for dst bucket item count to get to: 50000 curr count: 36244
2021/12/15 01:25:45 Waiting for dst bucket item count to get to: 50000 curr count: 36244
2021/12/15 01:25:50 Waiting for dst bucket item count to get to: 50000 curr count: 42488
2021/12/15 01:25:55 Waiting for dst bucket item count to get to: 50000 curr count: 42488
2021/12/15 01:25:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:25:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:25:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:26:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:26:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:26:10 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:26:15 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:26:20 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:26:25 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 01:26:25 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/15 01:26:25 No eventing-con process running
2021/12/15 01:26:25 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/15 01:26:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:26:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:26:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:26:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:26:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:26:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:26:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:26:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.91s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/15 01:26:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:26:59 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/15 01:26:59 No eventing-con process running
2021/12/15 01:26:59 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/15 01:27:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:27:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:27:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:27:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:27:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:27:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:27:17 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:27:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 01:27:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 01:27:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 01:27:33 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 01:27:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 01:27:38 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/15 01:27:38 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 01:27:38 Starting bucket ops to source bucket
2021/12/15 01:27:39 Removing node(s): [] from the cluster
2021/12/15 01:27:44 Rebalance progress: 12.5
2021/12/15 01:27:49 Rebalance progress: 12.5
2021/12/15 01:27:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:27:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:27:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:27:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:27:59 Rebalance progress: 12.5
2021/12/15 01:28:04 Rebalance progress: 12.5
2021/12/15 01:28:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:28:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:28:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:28:09 Waiting for src bucket item count to get to: 50000 curr count: 27170
2021/12/15 01:28:14 Waiting for src bucket item count to get to: 50000 curr count: 36101
2021/12/15 01:28:19 Waiting for src bucket item count to get to: 50000 curr count: 36101
2021/12/15 01:28:24 Waiting for src bucket item count to get to: 50000 curr count: 48838
2021/12/15 01:28:29 Waiting for src bucket item count to get to: 50000 curr count: 48838
2021/12/15 01:28:39 Waiting for dst bucket item count to get to: 50000 curr count: 43353
2021/12/15 01:28:44 Waiting for dst bucket item count to get to: 50000 curr count: 43353
2021/12/15 01:28:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:28:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:28:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:28:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:28:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:28:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:29:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:29:10 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:29:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 01:29:15 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/15 01:29:15 No eventing-con process running
2021/12/15 01:29:15 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/15 01:29:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:29:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:29:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:29:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:29:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:29:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.50s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/15 01:29:43 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 01:29:43 Removing node(s): [] from the cluster
2021/12/15 01:29:48 Rebalance progress: 100
2021/12/15 01:29:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:29:54 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/15 01:29:54 No eventing-con process running
2021/12/15 01:29:54 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/15 01:29:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:30:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:30:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:30:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:30:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:30:12 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:30:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 01:30:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 01:30:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 01:30:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 01:30:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 01:30:33 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/15 01:30:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:30:33 Starting bucket ops to source bucket
2021/12/15 01:30:38 Rebalance progress: 12.5
2021/12/15 01:30:43 Rebalance progress: 12.5
2021/12/15 01:30:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:30:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:30:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:30:48 Waiting for src bucket item count to get to: 50000 curr count: 7460
2021/12/15 01:30:53 Waiting for src bucket item count to get to: 50000 curr count: 21101
2021/12/15 01:30:58 Waiting for src bucket item count to get to: 50000 curr count: 21101
2021/12/15 01:31:03 Waiting for src bucket item count to get to: 50000 curr count: 32661
2021/12/15 01:31:08 Waiting for src bucket item count to get to: 50000 curr count: 32661
2021/12/15 01:31:13 Waiting for src bucket item count to get to: 50000 curr count: 43181
2021/12/15 01:31:18 Waiting for src bucket item count to get to: 50000 curr count: 43181
2021/12/15 01:31:23 Waiting for src bucket item count to get to: 50000 curr count: 43181
2021/12/15 01:31:33 Waiting for dst bucket item count to get to: 50000 curr count: 34580
2021/12/15 01:31:38 Waiting for dst bucket item count to get to: 50000 curr count: 34580
2021/12/15 01:31:44 Waiting for dst bucket item count to get to: 50000 curr count: 47247
2021/12/15 01:31:49 Waiting for dst bucket item count to get to: 50000 curr count: 47247
2021/12/15 01:31:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:31:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:31:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:31:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:31:59 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:32:04 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:32:09 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:32:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:32:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 01:32:19 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/15 01:32:19 No eventing-con process running
2021/12/15 01:32:19 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/15 01:32:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:32:31 Waiting for dst bucket item count to get to: 0 curr count: 33371
2021/12/15 01:32:36 Waiting for dst bucket item count to get to: 0 curr count: 33371
2021/12/15 01:32:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:32:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:32:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:32:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:32:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (189.32s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/15 01:32:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:32:58 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/15 01:32:58 No eventing-con process running
2021/12/15 01:32:58 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/15 01:32:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:33:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:33:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:33:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:33:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:33:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:33:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:33:16 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:33:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 01:33:22 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 01:33:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 01:33:32 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 01:33:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 01:33:37 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/15 01:33:37 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 01:33:37 Starting bucket ops to source bucket
2021/12/15 01:33:38 Removing node(s): [] from the cluster
2021/12/15 01:33:43 Rebalance progress: 12.5
2021/12/15 01:33:48 Rebalance progress: 12.5
2021/12/15 01:33:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:33:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 01:33:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:33:59 Rebalance progress: 12.5
2021/12/15 01:34:04 Rebalance progress: 12.5
2021/12/15 01:34:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:34:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:34:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:34:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:34:14 Rebalance progress: 12.5
2021/12/15 01:34:19 Rebalance progress: 12.5
2021/12/15 01:34:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:34:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:34:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:34:24 Waiting for src bucket item count to get to: 50000 curr count: 48129
2021/12/15 01:34:29 Waiting for src bucket item count to get to: 50000 curr count: 48129
2021/12/15 01:34:39 Waiting for dst bucket item count to get to: 50000 curr count: 46261
2021/12/15 01:34:44 Waiting for dst bucket item count to get to: 50000 curr count: 46261
2021/12/15 01:34:49 Waiting for dst bucket item count to get to: 50000 curr count: 49596
2021/12/15 01:34:54 Waiting for dst bucket item count to get to: 50000 curr count: 49596
2021/12/15 01:34:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:34:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:34:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:34:59 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:35:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:35:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:35:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:35:19 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:35:24 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 01:35:24 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/15 01:35:24 No eventing-con process running
2021/12/15 01:35:24 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/15 01:35:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:35:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:35:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:35:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:35:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:35:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (180.18s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/15 01:35:58 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:35:58 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/15 01:35:58 No eventing-con process running
2021/12/15 01:35:58 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/15 01:36:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:36:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:36:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:36:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:36:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:36:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:36:16 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:36:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 01:36:22 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 01:36:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 01:36:32 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 01:36:37 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 01:36:37 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/15 01:36:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:36:37 Starting bucket ops to source bucket
2021/12/15 01:36:38 Removing node(s): [] from the cluster
2021/12/15 01:36:43 Rebalance progress: 12.5
2021/12/15 01:36:48 Rebalance progress: 12.5
2021/12/15 01:36:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:36:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:36:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:36:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:36:58 Rebalance progress: 12.5
2021/12/15 01:37:03 Rebalance progress: 12.5
2021/12/15 01:37:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:37:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:37:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:37:08 Waiting for src bucket item count to get to: 50000 curr count: 27593
2021/12/15 01:37:13 Waiting for src bucket item count to get to: 50000 curr count: 37063
2021/12/15 01:37:18 Waiting for src bucket item count to get to: 50000 curr count: 37063
2021/12/15 01:37:23 Waiting for src bucket item count to get to: 50000 curr count: 49628
2021/12/15 01:37:28 Waiting for src bucket item count to get to: 50000 curr count: 49628
2021/12/15 01:37:38 Waiting for dst bucket item count to get to: 50000 curr count: 43038
2021/12/15 01:37:43 Waiting for dst bucket item count to get to: 50000 curr count: 43038
2021/12/15 01:37:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:37:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:37:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:37:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:37:54 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:37:59 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:38:04 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:38:09 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:38:14 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 01:38:14 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/15 01:38:14 No eventing-con process running
2021/12/15 01:38:14 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/15 01:38:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:38:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:38:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:38:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:38:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:38:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.69s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/15 01:38:42 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:38:43 Removing node(s): [] from the cluster
2021/12/15 01:38:48 Rebalance progress: 100
2021/12/15 01:38:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:38:53 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/15 01:38:53 No eventing-con process running
2021/12/15 01:38:53 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/15 01:38:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:39:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:39:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:39:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:39:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:39:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:39:11 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:39:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 01:39:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 01:39:23 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 01:39:28 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 01:39:33 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 01:39:33 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/15 01:39:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:39:33 Starting bucket ops to source bucket
2021/12/15 01:39:38 Rebalance progress: 12.5
2021/12/15 01:39:43 Rebalance progress: 12.5
2021/12/15 01:39:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:39:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:39:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:39:48 Waiting for src bucket item count to get to: 50000 curr count: 8535
2021/12/15 01:39:53 Waiting for src bucket item count to get to: 50000 curr count: 22381
2021/12/15 01:39:58 Waiting for src bucket item count to get to: 50000 curr count: 22381
2021/12/15 01:40:03 Waiting for src bucket item count to get to: 50000 curr count: 34039
2021/12/15 01:40:08 Waiting for src bucket item count to get to: 50000 curr count: 34039
2021/12/15 01:40:13 Waiting for src bucket item count to get to: 50000 curr count: 43756
2021/12/15 01:40:18 Waiting for src bucket item count to get to: 50000 curr count: 43756
2021/12/15 01:40:23 Waiting for src bucket item count to get to: 50000 curr count: 43756
2021/12/15 01:40:33 Waiting for dst bucket item count to get to: 50000 curr count: 34712
2021/12/15 01:40:38 Waiting for dst bucket item count to get to: 50000 curr count: 34712
2021/12/15 01:40:43 Waiting for dst bucket item count to get to: 50000 curr count: 49882
2021/12/15 01:40:48 Waiting for dst bucket item count to get to: 50000 curr count: 49882
2021/12/15 01:40:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:40:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:40:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:40:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:40:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:41:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:41:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:41:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:41:18 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 01:41:18 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/15 01:41:18 No eventing-con process running
2021/12/15 01:41:18 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/15 01:41:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:41:30 Waiting for dst bucket item count to get to: 0 curr count: 8647
2021/12/15 01:41:35 Waiting for dst bucket item count to get to: 0 curr count: 8647
2021/12/15 01:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:41:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:41:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (189.54s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/15 01:41:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:41:57 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/15 01:41:57 No eventing-con process running
2021/12/15 01:41:57 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/15 01:41:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:42:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:42:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:42:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:42:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:42:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:42:16 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:42:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 01:42:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 01:42:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 01:42:32 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 01:42:37 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 01:42:37 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/15 01:42:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:42:37 Starting bucket ops to source bucket
2021/12/15 01:42:37 Removing node(s): [] from the cluster
2021/12/15 01:42:42 Rebalance progress: 12.5
2021/12/15 01:42:47 Rebalance progress: 12.5
2021/12/15 01:42:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:42:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:42:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:42:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 01:42:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:42:58 Rebalance progress: 12.5
2021/12/15 01:43:03 Rebalance progress: 12.5
2021/12/15 01:43:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:43:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:43:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:43:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:43:13 Rebalance progress: 12.5
2021/12/15 01:43:18 Rebalance progress: 12.5
2021/12/15 01:43:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:43:23 Waiting for src bucket item count to get to: 50000 curr count: 47861
2021/12/15 01:43:28 Waiting for src bucket item count to get to: 50000 curr count: 47861
2021/12/15 01:43:38 Waiting for dst bucket item count to get to: 50000 curr count: 46946
2021/12/15 01:43:43 Waiting for dst bucket item count to get to: 50000 curr count: 46946
2021/12/15 01:43:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 01:43:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 01:43:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:43:48 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:43:53 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:43:59 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:44:04 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:44:09 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:44:14 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 01:44:14 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/15 01:44:14 No eventing-con process running
2021/12/15 01:44:14 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/15 01:44:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:44:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:44:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:44:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:44:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:44:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:44:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 01:44:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.34s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/15 01:44:47 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:44:47 App: bucket_op_on_update got un-deployed
2021/12/15 01:44:47 No eventing-con process running
2021/12/15 01:44:47 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:44:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:44:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:44:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:44:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:45:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:45:06 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:45:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 01:45:12 Starting bucket ops to source bucket
2021/12/15 01:45:17 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:45:22 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:45:27 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:45:27 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:45:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:45:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:45:32 Rebalance progress: 12.5
2021/12/15 01:45:37 Rebalance progress: 12.5
2021/12/15 01:45:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:45:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 01:45:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:45:48 Rebalance progress: 12.5
2021/12/15 01:45:53 Rebalance progress: 12.5
2021/12/15 01:45:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:45:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:45:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 01:45:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:46:03 Rebalance progress: 12.5
2021/12/15 01:46:08 Rebalance progress: 12.5
2021/12/15 01:46:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:46:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:46:18 Rebalance progress: 12.5
2021/12/15 01:46:23 Rebalance progress: 12.5
2021/12/15 01:46:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:46:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:46:34 Rebalance progress: 12.5
2021/12/15 01:46:39 Rebalance progress: 12.5
2021/12/15 01:46:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:46:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 01:46:49 Rebalance progress: 12.5
2021/12/15 01:46:54 Rebalance progress: 12.5
2021/12/15 01:46:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:46:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:46:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:47:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:47:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:47:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:47:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:47:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:47:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:47:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:47:29 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:47:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 01:47:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:47:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:47:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9953
2021/12/15 01:47:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9953
2021/12/15 01:47:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:47:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:47:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:47:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10656
2021/12/15 01:47:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.43s)
=== RUN   TestKVForceFailover
2021/12/15 01:48:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:48:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:48:13 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:48:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:48:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:48:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:48:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:48:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:48:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:48:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:48:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:48:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:48:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:48:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:48:32 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:48:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 01:48:33 Starting bucket ops to source bucket
2021/12/15 01:48:38 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:48:43 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:48:48 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:48:53 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:48:53 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:48:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:48:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:48:53 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/15 01:49:13 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/15 01:49:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 01:49:18 Rebalance progress: 12.5
2021/12/15 01:49:23 Rebalance progress: 12.5
2021/12/15 01:49:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:49:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 01:49:33 Rebalance progress: 12.5
2021/12/15 01:49:38 Rebalance progress: 12.5
2021/12/15 01:49:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:49:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:49:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:49:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:49:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:50:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:50:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:50:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:50:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:50:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:50:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 01:50:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:50:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:50:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7046
2021/12/15 01:50:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:50:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:50:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:50:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7046
2021/12/15 01:50:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.55s)
=== RUN   TestKVHardFailoverRecovery
2021/12/15 01:50:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:50:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:50:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:50:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:50:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:50:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:50:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:50:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:51:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:51:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:51:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:51:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 01:51:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 01:51:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:51:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:51:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:51:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:51:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:51:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:51:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:51:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:51:32 Rebalance progress: 12.5
2021/12/15 01:51:37 Rebalance progress: 12.5
2021/12/15 01:51:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:51:42 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/15 01:52:42 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/15 01:52:47 Rebalance progress: 12.5
2021/12/15 01:52:52 Rebalance progress: 12.5
2021/12/15 01:52:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:52:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:52:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:52:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:53:03 Rebalance progress: 12.5
2021/12/15 01:53:08 Rebalance progress: 12.5
2021/12/15 01:53:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:53:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:53:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:53:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:53:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:53:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:53:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:53:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:53:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:53:43 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:53:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 01:53:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:53:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:53:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12050
2021/12/15 01:53:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:53:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:54:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:54:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12050
2021/12/15 01:54:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.26s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/15 01:54:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:54:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:54:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:54:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:54:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:54:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:54:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:54:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:54:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:54:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:54:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:54:35 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:54:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 01:54:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 01:54:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:54:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:54:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:54:56 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:54:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:54:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:54:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:54:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 01:54:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 01:54:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:55:02 Rebalance progress: 12.5
2021/12/15 01:55:07 Rebalance progress: 12.5
2021/12/15 01:55:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:55:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:55:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:55:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 01:55:17 Rebalance progress: 12.5
2021/12/15 01:55:22 Rebalance progress: 12.5
2021/12/15 01:55:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:55:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:55:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:55:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:55:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:55:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:55:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:55:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:55:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:55:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:55:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:55:58 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:55:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 01:56:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:56:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:56:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:56:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:56:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:56:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4612
2021/12/15 01:56:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.89s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/15 01:56:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:56:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:56:26 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:56:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:56:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:56:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:56:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:56:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:56:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:56:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:56:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:56:40 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:56:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 01:56:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 01:56:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:56:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:57:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 01:57:01 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:57:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 01:57:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 01:57:06 Rebalance progress: 12.5
2021/12/15 01:57:11 Rebalance progress: 12.5
2021/12/15 01:57:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:57:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 01:57:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 01:57:22 Rebalance progress: 12.5
2021/12/15 01:57:27 Rebalance progress: 12.5
2021/12/15 01:57:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:57:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 01:57:37 Rebalance progress: 12.5
2021/12/15 01:57:42 Rebalance progress: 12.5
2021/12/15 01:57:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:57:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 01:57:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 01:57:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:57:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:58:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:58:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:58:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:58:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:58:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:58:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:58:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 01:58:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:58:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:58:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6079
2021/12/15 01:58:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:58:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:58:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:58:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6079
2021/12/15 01:58:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6079
2021/12/15 01:58:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:58:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:58:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:58:51 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:58:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:58:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:58:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:58:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:59:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.27s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/15 01:59:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 01:59:15 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 01:59:15 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:59:15 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 01:59:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:59:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:59:33 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 01:59:33 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 01:59:33 eventingSetup.go:46: No eventing-con process running
2021/12/15 01:59:33 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/15 01:59:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 01:59:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:47 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:59:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 01:59:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 01:59:48 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 01:59:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 01:59:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 01:59:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:00:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:00:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:00:09 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/15 02:00:14 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 02:00:19 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 02:00:24 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 02:00:24 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:00:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:00:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:00:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 02:00:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:00:30 Rebalance progress: 12.5
2021/12/15 02:00:35 Rebalance progress: 12.5
2021/12/15 02:00:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:00:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 02:00:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:00:45 Rebalance progress: 12.5
2021/12/15 02:00:50 Rebalance progress: 12.5
2021/12/15 02:00:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:00:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 02:00:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:01:01 Rebalance progress: 12.5
2021/12/15 02:01:06 Rebalance progress: 12.5
2021/12/15 02:01:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:01:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 02:01:16 Rebalance progress: 12.5
2021/12/15 02:01:21 Rebalance progress: 12.5
2021/12/15 02:01:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:01:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 02:01:31 Rebalance progress: 12.5
2021/12/15 02:01:36 Rebalance progress: 12.5
2021/12/15 02:01:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:01:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 02:01:46 Rebalance progress: 12.5
2021/12/15 02:01:51 Rebalance progress: 12.5
2021/12/15 02:01:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:01:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:01:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:02:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:02:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:02:32 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:02:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 02:02:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:02:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:02:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12220
2021/12/15 02:02:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:02:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:02:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:02:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12220
2021/12/15 02:02:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:02:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:03:02 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:03:07 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:03:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:03:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:03:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:03:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:03:27 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 02:03:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:03:28 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/15 02:03:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:03:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:03:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:03:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:03:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:03:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (276.45s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/15 02:03:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:03:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:03:51 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:03:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 02:03:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:03:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:03:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:04:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:04:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:04:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:04:10 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:04:10 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 02:04:10 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:04:10 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/15 02:04:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:04:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:04:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:04:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:04:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:04:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:04:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:04:23 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:04:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:04:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:04:25 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:04:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 02:04:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 02:04:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:04:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:04:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:04:46 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/15 02:04:51 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 02:04:56 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 02:05:01 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 02:05:01 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:05:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 02:05:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:05:06 Rebalance progress: 12.5
2021/12/15 02:05:11 Rebalance progress: 12.5
2021/12/15 02:05:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:05:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 02:05:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 02:05:22 Rebalance progress: 12.5
2021/12/15 02:05:27 Rebalance progress: 12.5
2021/12/15 02:05:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:05:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 02:05:37 Rebalance progress: 12.5
2021/12/15 02:05:42 Rebalance progress: 12.5
2021/12/15 02:05:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:05:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:05:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:05:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:06:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:06:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:06:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:06:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:06:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:06:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:06:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 02:06:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:06:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:06:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 673
2021/12/15 02:06:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 673
2021/12/15 02:06:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:06:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:06:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:06:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7598
2021/12/15 02:06:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:06:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:06:53 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:06:58 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:07:03 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:07:08 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:07:13 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:07:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 02:07:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 02:07:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:07:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/15 02:07:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:07:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:07:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:07:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:07:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:07:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.80s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 02:07:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:07:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:07:42 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:07:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 02:07:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:07:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:07:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:07:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:07:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:07:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:08:00 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:08:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 02:08:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 02:08:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:08:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:08:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:08:22 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:08:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 02:08:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:08:42 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:08:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:08:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:08:58 Rebalance progress: 12.5
2021/12/15 02:09:03 Rebalance progress: 12.5
2021/12/15 02:09:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:09:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:09:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:09:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 02:09:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 02:09:13 Rebalance progress: 12.5
2021/12/15 02:09:18 Rebalance progress: 12.5
2021/12/15 02:09:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:09:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 02:09:29 Rebalance progress: 12.5
2021/12/15 02:09:34 Rebalance progress: 12.5
2021/12/15 02:09:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:09:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:09:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:09:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:09:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:09:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:09:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:09:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:10:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:10:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:10:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:10:09 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:10:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 02:10:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:10:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:10:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7055
2021/12/15 02:10:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7055
2021/12/15 02:10:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:10:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:10:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:10:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9180
2021/12/15 02:10:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.64s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/15 02:10:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:10:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:10:48 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:10:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 02:10:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:10:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:10:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:10:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:11:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:11:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:11:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 02:11:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 02:11:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:11:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:11:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:11:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:11:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 02:11:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:11:33 Rebalance progress: 12.5
2021/12/15 02:11:38 Rebalance progress: 12.5
2021/12/15 02:11:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:11:43 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:11:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:11:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:11:58 Rebalance progress: 12.5
2021/12/15 02:12:03 Rebalance progress: 12.5
2021/12/15 02:12:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:12:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:12:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:12:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 02:12:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 02:12:13 Rebalance progress: 12.5
2021/12/15 02:12:18 Rebalance progress: 12.5
2021/12/15 02:12:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:12:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 02:12:28 Rebalance progress: 12.5
2021/12/15 02:12:33 Rebalance progress: 12.5
2021/12/15 02:12:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:12:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:12:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:12:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:12:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:12:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:12:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:12:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:13:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:13:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:13:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:13:09 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:13:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 02:13:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:13:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:13:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5042
2021/12/15 02:13:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5042
2021/12/15 02:13:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:13:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:13:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:13:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8596
2021/12/15 02:13:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.78s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/15 02:13:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 02:13:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 02:13:48 Rebalance progress: 100
2021/12/15 02:13:48 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:14:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:14:18 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 02:14:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 02:14:24 Removing node(s): [] from the cluster
2021/12/15 02:14:24 Starting bucket ops to source bucket
2021/12/15 02:14:29 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/15 02:14:34 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:14:39 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 02:14:39 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 02:14:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 02:14:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:14:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:14:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:14:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:15:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 02:15:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 02:15:04 eventingSetup.go:46: No eventing-con process running
2021/12/15 02:15:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 02:15:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:15:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:15:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/15 02:15:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 02:15:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 02:15:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 02:15:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/15 02:15:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.50s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12154.471s

Rebalance tests took 3 hours, 22 minutes and 37 seconds.

Testing Succeeded


Version: versions-14.12.2021-18.35.cfg
Build Log: make-14.12.2021-18.35.log
Server Log: logs-14.12.2021-18.35.tar.gz

Finished