Building

Started building at 2021/12/16 10:17:41
Calculating base
Updating mirror
Basing run on 7.1.0-1924 6f36c017e
Updating tree for run 16.12.2021-10.17
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f0d7a1e, changes since last good build: none
kv_engine is at 0811a49, changes since last good build: none
ns_server is at 009dd9f, changes since last good build: 
 009dd9f62 MB-50011 stats_interface filter non-existent buckets
 4bcdf5b19 MB-49765 remove some unused pre 6.5 code
couchstore is at 6eea212, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at c1e9d9b, changes since last good build: none
Switching eventing to unstable
eventing is at 7f52754, changes since last good build: 
 7f527545 MB-46304: Add functionality to disable curl feature
 d26894a7 MB-49547: Refresh bucket info from streaming endpoint instance
Switching eventing-ee to unstable
eventing-ee is at 3f826e0, changes since last good build: 
 3f826e0 MB-46304: Throw runtime exception when curl feature is disabled
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable dace8039] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 3ab7cc2f] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 1734aaec] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 3b5b1b10] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 167196 at refs/changes/96/167196/7
Total 29 (delta 25), reused 29 (delta 25)
[unstable 89d04f3b] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/5
Total 44 (delta 38), reused 44 (delta 38)
[unstable b0fc1f35] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/4
Total 52 (delta 44), reused 52 (delta 44)
[unstable fe466301] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/16 10:33:16
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/16 10:34:48 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:59774->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:59774->127.0.0.1:9000: read: connection reset by peer
2021/12/16 10:34:57 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 10:35:03 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/16 10:35:03 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 10:35:10 Removing node(s): [] from the cluster
2021/12/16 10:35:15 Rebalance progress: 0
2021/12/16 10:35:20 Rebalance progress: 2.043547453703704
2021/12/16 10:35:25 Rebalance progress: 6.655092592592593
2021/12/16 10:35:30 Rebalance progress: 10.05497685185185
2021/12/16 10:35:35 Rebalance progress: 14.72077546296296
2021/12/16 10:35:40 Rebalance progress: 18.51851851851852
2021/12/16 10:35:45 Rebalance progress: 22.65986689814815
2021/12/16 10:35:50 Rebalance progress: 27.25332754629629
2021/12/16 10:35:55 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/16 10:36:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:36:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:36:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:36:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:37:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:37:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:37:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:37:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:38:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:38:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:38:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:38:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:39:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:39:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:39:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:39:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:40:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:40:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:40:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:40:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:41:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:41:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:41:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:41:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 10:42:01 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:42:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:42:07 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 10:42:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 10:42:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 10:42:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 10:42:22 App: TestAdvancedGetOps got deployed
2021/12/16 10:42:22 Starting bucket ops to source bucket
2021/12/16 10:42:27 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 10:42:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 10:42:37 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 10:42:37 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 10:42:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:42:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:42:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:42:42 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 10:42:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 10:42:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 10:42:57 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 10:42:57 App: TestAdvancedGetOps got un-deployed
2021/12/16 10:42:57 No eventing-con process running
2021/12/16 10:42:57 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/16 10:42:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:43:09 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/16 10:43:14 Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/16 10:43:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:43:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:43:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (445.26s)
=== RUN   TestAdvancedInsertOps
2021/12/16 10:43:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:43:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:43:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:43:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:43:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:43:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:43:47 App: TestAdvancedInsertOps got deployed
2021/12/16 10:43:47 Starting bucket ops to source bucket
2021/12/16 10:43:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 10:43:58 Waiting for dst bucket item count to get to: 1024 curr count: 720
2021/12/16 10:44:03 Waiting for dst bucket item count to get to: 1024 curr count: 720
2021/12/16 10:44:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 10:44:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:44:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:44:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:44:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:44:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:44:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:44:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:44:23 App: TestAdvancedInsertOps got un-deployed
2021/12/16 10:44:23 No eventing-con process running
2021/12/16 10:44:23 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/16 10:44:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:44:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:44:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:44:46 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:44:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:44:46 Testing insert operation on source bucket
2021/12/16 10:44:46 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:44:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:44:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:44:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:45:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:45:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 10:45:08 App: TestAdvancedInsertOps got deployed
2021/12/16 10:45:08 Starting bucket ops to source bucket
2021/12/16 10:45:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 10:45:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 10:45:24 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 10:45:24 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 10:45:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:45:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:45:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:45:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:45:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:45:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:45:44 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 10:45:44 App: TestAdvancedInsertOps got un-deployed
2021/12/16 10:45:44 No eventing-con process running
2021/12/16 10:45:44 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/16 10:45:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:45:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/16 10:45:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:45:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:46:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.64s)
=== RUN   TestAdvancedUpsertOps
2021/12/16 10:46:02 Starting bucket ops to source bucket
2021/12/16 10:46:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:46:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:46:09 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:46:14 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:46:19 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:46:24 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:46:24 App: TestAdvancedUpsertOps got deployed
2021/12/16 10:46:24 Starting bucket ops to source bucket
2021/12/16 10:46:30 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:46:35 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 10:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:46:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:46:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:46:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:46:40 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:46:45 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:46:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:46:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:46:55 App: TestAdvancedUpsertOps got un-deployed
2021/12/16 10:46:55 No eventing-con process running
2021/12/16 10:46:55 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/16 10:46:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:47:07 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:47:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:47:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:47:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:47:14 Testing upsert operation on source bucket
2021/12/16 10:47:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:47:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:47:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:47:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:47:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:47:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 10:47:35 App: TestAdvancedUpsertOps got deployed
2021/12/16 10:47:35 Starting bucket ops to source bucket
2021/12/16 10:47:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:47:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:47:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:47:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:47:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:47:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:47:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:47:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 10:47:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/16 10:47:56 No eventing-con process running
2021/12/16 10:47:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/16 10:47:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:48:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:48:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.01s)
=== RUN   TestAdvancedReplaceOps
2021/12/16 10:48:10 Starting bucket ops to source bucket
2021/12/16 10:48:11 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:48:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:48:17 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:48:22 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:48:27 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:48:32 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:48:32 App: TestAdvancedReplaceOps got deployed
2021/12/16 10:48:32 Starting bucket ops to source bucket
2021/12/16 10:48:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:48:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:48:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:48:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:48:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:48:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:48:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:48:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:49:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:49:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/16 10:49:04 No eventing-con process running
2021/12/16 10:49:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/16 10:49:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:49:15 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 10:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:49:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:49:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:49:22 Testing upsert operation on source bucket
2021/12/16 10:49:22 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:49:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:49:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:49:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:49:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:49:43 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 10:49:43 App: TestAdvancedReplaceOps got deployed
2021/12/16 10:49:43 Starting bucket ops to source bucket
2021/12/16 10:49:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:49:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:49:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:49:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:49:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:49:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:50:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:50:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 10:50:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/16 10:50:05 No eventing-con process running
2021/12/16 10:50:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/16 10:50:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:50:17 Waiting for dst bucket item count to get to: 0 curr count: 170
2021/12/16 10:50:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:50:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:50:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.99s)
=== RUN   TestAdvancedDeleteOps
2021/12/16 10:50:23 Starting bucket ops to source bucket
2021/12/16 10:50:24 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:50:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:50:30 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:50:35 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:50:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:50:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:50:45 App: TestAdvancedDeleteOps got deployed
2021/12/16 10:50:45 Starting bucket ops to source bucket
2021/12/16 10:50:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:50:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 10:50:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:50:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:50:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:50:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:51:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:51:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:51:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:51:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:51:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/16 10:51:15 No eventing-con process running
2021/12/16 10:51:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/16 10:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:51:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:51:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:51:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:51:33 Testing delete operation on source bucket
2021/12/16 10:51:33 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:51:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/16 10:51:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:51:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:51:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:51:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 10:51:55 App: TestAdvancedDeleteOps got deployed
2021/12/16 10:51:55 Starting bucket ops to source bucket
2021/12/16 10:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:51:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:51:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:51:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:52:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:52:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:52:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:52:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 10:52:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/16 10:52:15 No eventing-con process running
2021/12/16 10:52:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/16 10:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:52:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:52:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.76s)
=== RUN   TestEnoentAdvancedGet
2021/12/16 10:52:29 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:52:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/16 10:52:35 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:52:40 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:52:45 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:52:50 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:52:50 App: TestEnoentAdvancedGet got deployed
2021/12/16 10:52:50 Starting bucket ops to source bucket
2021/12/16 10:52:55 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 10:53:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 10:53:00 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 10:53:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:53:00 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:53:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:53:05 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:53:10 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:53:15 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:53:20 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:53:20 App: TestEnoentAdvancedGet got un-deployed
2021/12/16 10:53:20 No eventing-con process running
2021/12/16 10:53:20 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/16 10:53:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:53:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:53:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:53:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:53:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:53:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 10:53:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:53:54 Testing get enoent operation on source bucket
2021/12/16 10:53:54 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:53:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/16 10:54:00 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:54:05 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:54:10 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:54:15 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 10:54:15 App: TestEnoentAdvancedGet got deployed
2021/12/16 10:54:15 Starting bucket ops to source bucket
2021/12/16 10:54:20 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 10:54:20 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 10:54:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:54:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:54:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:54:25 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:54:30 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:54:35 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:54:40 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 10:54:40 App: TestEnoentAdvancedGet got un-deployed
2021/12/16 10:54:40 No eventing-con process running
2021/12/16 10:54:40 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/16 10:54:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:54:52 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 10:54:57 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 10:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:55:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.91s)
=== RUN   TestEnoentAdvancedReplace
2021/12/16 10:55:04 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:55:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 10:55:10 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:55:15 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:55:20 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:55:25 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:55:25 App: TestEnoentAdvancedReplace got deployed
2021/12/16 10:55:25 Starting bucket ops to source bucket
2021/12/16 10:55:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 10:55:30 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 10:55:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:55:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:55:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:55:35 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:55:40 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:55:45 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:55:50 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:55:50 App: TestEnoentAdvancedReplace got un-deployed
2021/12/16 10:55:50 No eventing-con process running
2021/12/16 10:55:50 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/16 10:55:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:56:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:56:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:56:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:56:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:56:19 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/16 10:56:24 Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/16 10:56:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:56:24 Testing get enoent operation on source bucket
2021/12/16 10:56:24 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:56:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 10:56:30 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:56:35 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:56:40 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:56:45 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 10:56:45 App: TestEnoentAdvancedReplace got deployed
2021/12/16 10:56:45 Starting bucket ops to source bucket
2021/12/16 10:56:50 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 10:56:50 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 10:56:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:56:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:56:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:56:55 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:57:00 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:57:05 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:57:10 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 10:57:10 App: TestEnoentAdvancedReplace got un-deployed
2021/12/16 10:57:10 No eventing-con process running
2021/12/16 10:57:10 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/16 10:57:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:57:22 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 10:57:27 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 10:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.95s)
=== RUN   TestEnoentAdvancedDelete
2021/12/16 10:57:34 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:57:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 10:57:40 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:57:45 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:57:50 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:57:55 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:57:55 App: TestEnoentAdvancedDelete got deployed
2021/12/16 10:57:55 Starting bucket ops to source bucket
2021/12/16 10:58:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 10:58:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 10:58:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 10:58:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:58:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:58:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:58:10 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:58:15 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:58:20 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:58:25 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:58:25 App: TestEnoentAdvancedDelete got un-deployed
2021/12/16 10:58:25 No eventing-con process running
2021/12/16 10:58:25 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/16 10:58:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:58:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 10:58:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:58:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:58:49 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/16 10:58:54 Waiting for dst bucket item count to get to: 0 curr count: 48
2021/12/16 10:58:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:58:54 Testing get enoent operation on source bucket
2021/12/16 10:58:54 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:58:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 10:59:00 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:59:05 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:59:10 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:59:15 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 10:59:15 App: TestEnoentAdvancedDelete got deployed
2021/12/16 10:59:15 Starting bucket ops to source bucket
2021/12/16 10:59:20 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 10:59:20 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 10:59:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 10:59:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 10:59:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:59:25 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:59:30 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:59:35 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:59:40 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 10:59:40 App: TestEnoentAdvancedDelete got un-deployed
2021/12/16 10:59:40 No eventing-con process running
2021/12/16 10:59:40 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/16 10:59:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 10:59:52 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 10:59:57 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 10:59:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:59:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:00:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.09s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/16 11:00:04 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:00:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/16 11:00:10 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:00:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:00:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:00:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:00:25 App: TestAdvancedInsertKeyExist got deployed
2021/12/16 11:00:25 Starting bucket ops to source bucket
2021/12/16 11:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:00:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:00:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:00:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:00:31 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:00:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:00:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:00:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:00:46 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/16 11:00:46 No eventing-con process running
2021/12/16 11:00:46 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/16 11:00:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:00:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:00:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:01:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:01:00 Pumping document in the destination bucket
2021/12/16 11:01:00 Starting bucket ops to source bucket
2021/12/16 11:01:00 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:01:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/16 11:01:06 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:01:11 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:01:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:01:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 11:01:21 App: TestAdvancedInsertKeyExist got deployed
2021/12/16 11:01:21 Starting bucket ops to source bucket
2021/12/16 11:01:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:01:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:01:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:01:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:01:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:01:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:01:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:01:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:01:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:01:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 11:01:51 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/16 11:01:51 No eventing-con process running
2021/12/16 11:01:51 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/16 11:01:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:02:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:02:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:02:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:02:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:02:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (131.00s)
=== RUN   TestExipryGet
2021/12/16 11:02:15 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:02:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/16 11:02:21 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:02:26 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:02:31 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:02:36 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:02:36 App: TestExipryGet got deployed
2021/12/16 11:02:36 Starting bucket ops to source bucket
2021/12/16 11:03:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:03:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:03:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:03:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:03:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:03:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:03:32 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:03:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:03:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:03:42 App: TestExipryGet got un-deployed
2021/12/16 11:03:42 No eventing-con process running
2021/12/16 11:03:42 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/16 11:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:03:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:03:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:03:56 Pumping document in the destination bucket for expiry get
2021/12/16 11:03:56 Starting bucket ops to source bucket
2021/12/16 11:03:56 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:03:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/16 11:04:02 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:04:07 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:04:12 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:04:17 Waiting for app: TestExipryGet to get deployed
2021/12/16 11:04:17 App: TestExipryGet got deployed
2021/12/16 11:04:17 Starting bucket ops to source bucket
2021/12/16 11:05:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:05:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:05:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:05:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:05:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:05:07 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:05:12 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:05:17 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:05:22 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 11:05:22 App: TestExipryGet got un-deployed
2021/12/16 11:05:22 No eventing-con process running
2021/12/16 11:05:22 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/16 11:05:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:05:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:05:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:05:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.80s)
=== RUN   TestExipryInsert
2021/12/16 11:05:46 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:05:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/16 11:05:52 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:05:57 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:06:02 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:06:07 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:06:07 App: TestExipryInsert got deployed
2021/12/16 11:06:07 Starting bucket ops to source bucket
2021/12/16 11:06:52 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 11:06:57 Waiting for dst bucket item count to get to: 0 curr count: 44
2021/12/16 11:07:02 Waiting for dst bucket item count to get to: 0 curr count: 44
2021/12/16 11:07:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:07:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:07:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:07:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:07:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:07:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:07:17 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:07:22 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:07:22 App: TestExipryInsert got un-deployed
2021/12/16 11:07:22 No eventing-con process running
2021/12/16 11:07:22 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/16 11:07:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:07:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:07:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:07:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:07:36 Testing on the destination bucket for expiry insert
2021/12/16 11:07:36 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:07:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/16 11:07:43 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:07:48 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:07:53 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:07:58 Waiting for app: TestExipryInsert to get deployed
2021/12/16 11:07:58 App: TestExipryInsert got deployed
2021/12/16 11:07:58 Starting bucket ops to source bucket
2021/12/16 11:08:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:08:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:08:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:08:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:08:48 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:08:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:08:53 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:08:58 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:09:03 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:09:08 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 11:09:08 App: TestExipryInsert got un-deployed
2021/12/16 11:09:08 No eventing-con process running
2021/12/16 11:09:08 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/16 11:09:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:09:20 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:09:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:09:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:09:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (225.59s)
=== RUN   TestExipryReplace
2021/12/16 11:09:31 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:09:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 11:09:38 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:09:43 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:09:48 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:09:53 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:09:53 App: TestExipryReplace got deployed
2021/12/16 11:09:53 Starting bucket ops to source bucket
2021/12/16 11:10:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:10:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:10:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:10:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:10:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:10:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:10:48 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:10:53 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:10:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:11:03 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:11:03 App: TestExipryReplace got un-deployed
2021/12/16 11:11:03 No eventing-con process running
2021/12/16 11:11:03 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/16 11:11:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:11:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:11:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:11:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:11:17 Testing on the destination bucket for expiry insert
2021/12/16 11:11:17 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:11:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 11:11:23 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:11:28 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:11:33 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:11:38 Waiting for app: TestExipryReplace to get deployed
2021/12/16 11:11:38 App: TestExipryReplace got deployed
2021/12/16 11:11:38 Starting bucket ops to source bucket
2021/12/16 11:12:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:12:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:12:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:12:28 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:12:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:12:33 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:12:38 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:12:43 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:12:48 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 11:12:48 App: TestExipryReplace got un-deployed
2021/12/16 11:12:48 No eventing-con process running
2021/12/16 11:12:48 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/16 11:12:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:13:00 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/16 11:13:05 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/16 11:13:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:13:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:13:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.61s)
=== RUN   TestExipryUpsert
2021/12/16 11:13:12 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:13:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/16 11:13:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:13:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:13:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:13:33 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:13:33 App: TestExipryUpsert got deployed
2021/12/16 11:13:33 Starting bucket ops to source bucket
2021/12/16 11:14:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:14:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:14:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:14:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:14:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:14:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:14:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:14:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:14:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:14:44 App: TestExipryUpsert got un-deployed
2021/12/16 11:14:44 No eventing-con process running
2021/12/16 11:14:44 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/16 11:14:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:14:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:14:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:14:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:14:58 Testing on the destination bucket for expiry insert
2021/12/16 11:14:58 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:14:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/16 11:15:04 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:15:09 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:15:14 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:15:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 11:15:19 App: TestExipryUpsert got deployed
2021/12/16 11:15:19 Starting bucket ops to source bucket
2021/12/16 11:16:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:16:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:16:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:16:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:16:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:16:09 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:16:14 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:16:19 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:16:24 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 11:16:24 App: TestExipryUpsert got un-deployed
2021/12/16 11:16:24 No eventing-con process running
2021/12/16 11:16:24 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/16 11:16:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:16:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:16:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.56s)
=== RUN   TestCasReplace
2021/12/16 11:16:43 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:16:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 11:16:49 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:16:54 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:16:59 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:17:04 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:17:04 App: TestCasReplace got deployed
2021/12/16 11:17:04 Starting bucket ops to source bucket
2021/12/16 11:17:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:17:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:17:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:17:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:17:09 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:17:14 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:17:19 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:17:24 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:17:24 App: TestCasReplace got un-deployed
2021/12/16 11:17:24 No eventing-con process running
2021/12/16 11:17:24 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/16 11:17:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:17:36 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/16 11:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:17:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:17:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:17:43 Testing on the destination bucket for upsert with cas
2021/12/16 11:17:43 Starting bucket ops to source bucket
2021/12/16 11:17:43 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:17:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 11:17:49 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:17:54 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:17:59 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:18:04 Waiting for app: TestCasReplace to get deployed
2021/12/16 11:18:04 App: TestCasReplace got deployed
2021/12/16 11:18:04 Starting bucket ops to source bucket
2021/12/16 11:18:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:18:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:18:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:18:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:18:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:18:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:18:19 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:18:24 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:18:29 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:18:34 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 11:18:34 App: TestCasReplace got un-deployed
2021/12/16 11:18:34 No eventing-con process running
2021/12/16 11:18:34 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/16 11:18:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:18:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:18:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:18:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:18:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.03s)
=== RUN   TestCasDelete
2021/12/16 11:18:53 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:18:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 11:18:59 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:19:04 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:19:09 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:19:14 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:19:14 App: TestCasDelete got deployed
2021/12/16 11:19:14 Starting bucket ops to source bucket
2021/12/16 11:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:19:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:19:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:19:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:19:19 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:19:24 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:19:29 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:19:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:19:34 App: TestCasDelete got un-deployed
2021/12/16 11:19:34 No eventing-con process running
2021/12/16 11:19:34 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/16 11:19:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:19:46 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/16 11:19:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:19:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:19:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:19:53 Testing on the destination bucket for delete with cas
2021/12/16 11:19:53 Starting bucket ops to source bucket
2021/12/16 11:19:53 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:19:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 11:19:59 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:20:04 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:20:09 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:20:14 Waiting for app: TestCasDelete to get deployed
2021/12/16 11:20:14 App: TestCasDelete got deployed
2021/12/16 11:20:14 Starting bucket ops to source bucket
2021/12/16 11:20:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:20:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:20:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:20:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:20:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 11:20:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:20:29 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:20:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:20:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:20:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 11:20:44 App: TestCasDelete got un-deployed
2021/12/16 11:20:44 No eventing-con process running
2021/12/16 11:20:44 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/16 11:20:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:20:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:20:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:20:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:21:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.18s)
=== RUN   TestCountersIncrement
2021/12/16 11:21:03 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 11:21:09 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 11:21:09 Removing node(s): [] from the cluster
2021/12/16 11:21:14 Rebalance progress: 100
2021/12/16 11:21:14 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:21:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/16 11:21:21 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:21:26 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:21:31 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:21:36 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:21:36 App: TestCountersIncrement got deployed
2021/12/16 11:21:36 Starting bucket ops to source bucket
2021/12/16 11:21:42 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 11:21:47 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/16 11:21:47 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 11:21:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:21:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:21:52 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:21:57 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:22:02 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:22:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:22:07 App: TestCountersIncrement got un-deployed
2021/12/16 11:22:07 No eventing-con process running
2021/12/16 11:22:07 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/16 11:22:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:22:19 Waiting for dst bucket item count to get to: 0 curr count: 920
2021/12/16 11:22:24 Waiting for dst bucket item count to get to: 0 curr count: 407
2021/12/16 11:22:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:22:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:22:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 11:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:22:35 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:22:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/16 11:22:42 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:22:47 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:22:52 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:22:57 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 11:22:57 App: TestCountersIncrement got deployed
2021/12/16 11:22:57 Starting bucket ops to source bucket
2021/12/16 11:23:03 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 11:23:08 Waiting for dst bucket item count to get to: 2048 curr count: 1548
2021/12/16 11:23:13 Waiting for dst bucket item count to get to: 2048 curr count: 1548
2021/12/16 11:23:13 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 11:23:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:23:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:23:18 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:23:23 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:23:28 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:23:33 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 11:23:33 App: TestCountersIncrement got un-deployed
2021/12/16 11:23:33 No eventing-con process running
2021/12/16 11:23:33 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/16 11:23:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:23:45 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/16 11:23:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:23:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:23:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 11:23:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:23:56 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/16 11:24:02 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.81s)
=== RUN   TestCountersDecrement
2021/12/16 11:24:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 11:24:02 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 11:24:02 Removing node(s): [] from the cluster
2021/12/16 11:24:07 Rebalance progress: 100
2021/12/16 11:24:07 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:24:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/16 11:24:13 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:24:18 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:24:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:24:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:24:28 App: TestCountersDecrement got deployed
2021/12/16 11:24:28 Starting bucket ops to source bucket
2021/12/16 11:24:35 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 11:24:40 Waiting for dst bucket item count to get to: 1024 curr count: 331
2021/12/16 11:24:45 Waiting for dst bucket item count to get to: 1024 curr count: 331
2021/12/16 11:24:45 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 11:24:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:24:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:24:50 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:24:55 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:25:00 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:25:05 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:25:05 App: TestCountersDecrement got un-deployed
2021/12/16 11:25:05 No eventing-con process running
2021/12/16 11:25:05 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/16 11:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:25:17 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/16 11:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:25:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:25:28 Waiting for dst bucket item count to get to: 0 curr count: 888
2021/12/16 11:25:33 Waiting for dst bucket item count to get to: 0 curr count: 376
2021/12/16 11:25:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:25:34 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:25:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/16 11:25:40 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:25:45 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:25:50 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:25:55 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 11:25:55 App: TestCountersDecrement got deployed
2021/12/16 11:25:55 Starting bucket ops to source bucket
2021/12/16 11:26:01 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 11:26:06 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 11:26:06 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 11:26:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:26:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:26:11 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:26:16 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:26:21 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:26:26 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 11:26:26 App: TestCountersDecrement got un-deployed
2021/12/16 11:26:26 No eventing-con process running
2021/12/16 11:26:26 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/16 11:26:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:26:38 Waiting for dst bucket item count to get to: 0 curr count: 1267
2021/12/16 11:26:43 Waiting for dst bucket item count to get to: 0 curr count: 243
2021/12/16 11:26:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:26:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:26:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 11:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:26:55 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/16 11:27:00 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.20s)
=== RUN   TestCacheGet
2021/12/16 11:27:00 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:27:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/16 11:27:06 Waiting for app: TestCacheGet to get deployed
2021/12/16 11:27:11 Waiting for app: TestCacheGet to get deployed
2021/12/16 11:27:16 Waiting for app: TestCacheGet to get deployed
2021/12/16 11:27:21 Waiting for app: TestCacheGet to get deployed
2021/12/16 11:27:21 App: TestCacheGet got deployed
2021/12/16 11:27:21 Starting bucket ops to source bucket
2021/12/16 11:27:26 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 11:27:31 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 11:27:36 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/16 11:27:41 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/16 11:27:46 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/16 11:27:51 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/16 11:27:51 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/16 11:27:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:27:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:27:56 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 11:28:01 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 11:28:06 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 11:28:11 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 11:28:11 App: TestCacheGet got un-deployed
2021/12/16 11:28:11 No eventing-con process running
2021/12/16 11:28:11 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/16 11:28:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:28:23 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/16 11:28:28 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/16 11:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:28:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:28:40 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/16 11:28:45 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/16 11:28:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.28s)
=== RUN   TestCacheRYOW
2021/12/16 11:28:45 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:28:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/16 11:28:51 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 11:28:56 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 11:29:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 11:29:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 11:29:06 App: TestCacheRYOW got deployed
2021/12/16 11:29:06 Starting bucket ops to source bucket
2021/12/16 11:29:11 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/16 11:29:11 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/16 11:29:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:29:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:29:17 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 11:29:22 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 11:29:27 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 11:29:32 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 11:29:32 App: TestCacheRYOW got un-deployed
2021/12/16 11:29:32 No eventing-con process running
2021/12/16 11:29:32 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/16 11:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:29:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:29:48 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/16 11:29:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:29:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:30:00 Waiting for dst bucket item count to get to: 0 curr count: 36
2021/12/16 11:30:05 Waiting for dst bucket item count to get to: 0 curr count: 36
2021/12/16 11:30:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.19s)
=== RUN   TestCacheExpiry
2021/12/16 11:30:05 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:30:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/16 11:30:11 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 11:30:16 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 11:30:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 11:30:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 11:30:26 App: TestCacheExpiry got deployed
2021/12/16 11:30:26 Starting bucket ops to source bucket
2021/12/16 11:30:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:30:37 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 11:30:42 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 11:30:47 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 11:30:52 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 11:30:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:30:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:30:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:30:57 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 11:31:02 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 11:31:07 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 11:31:12 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 11:31:12 App: TestCacheExpiry got un-deployed
2021/12/16 11:31:12 No eventing-con process running
2021/12/16 11:31:12 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/16 11:31:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:31:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:31:29 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/16 11:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:31:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:31:40 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/16 11:31:45 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/16 11:31:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.18s)
=== RUN   TestCacheExpiryCustom
2021/12/16 11:31:45 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:31:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/16 11:31:52 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 11:31:57 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 11:32:02 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 11:32:07 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 11:32:07 App: TestCacheExpiryCustom got deployed
2021/12/16 11:32:07 Starting bucket ops to source bucket
2021/12/16 11:32:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:32:17 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 11:32:22 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/16 11:32:27 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 11:32:32 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/16 11:32:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:32:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:32:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:32:37 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 11:32:42 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 11:32:47 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 11:32:52 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 11:32:52 App: TestCacheExpiryCustom got un-deployed
2021/12/16 11:32:52 No eventing-con process running
2021/12/16 11:32:52 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/16 11:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:33:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:33:09 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/16 11:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:33:21 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/16 11:33:26 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/16 11:33:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.24s)
=== RUN   TestCacheOverflowCustom
2021/12/16 11:33:26 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:33:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/16 11:33:32 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 11:33:37 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 11:33:42 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 11:33:47 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 11:33:47 App: TestCacheOverflowCustom got deployed
2021/12/16 11:33:47 Starting bucket ops to source bucket
2021/12/16 11:33:52 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/16 11:33:57 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/16 11:34:02 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/16 11:34:07 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/16 11:34:12 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/16 11:34:12 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/16 11:34:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:34:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:34:17 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 11:34:22 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 11:34:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 11:34:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 11:34:32 App: TestCacheOverflowCustom got un-deployed
2021/12/16 11:34:32 No eventing-con process running
2021/12/16 11:34:32 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/16 11:34:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:34:44 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/16 11:34:49 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/16 11:34:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:34:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:35:01 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/16 11:35:06 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/16 11:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.30s)
=== RUN   TestStrictMode
2021/12/16 11:35:06 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:35:07 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.07s)
=== RUN   TestDataTypes
2021/12/16 11:35:07 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:35:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/16 11:35:13 Waiting for app: TestDataTypes to get deployed
2021/12/16 11:35:18 Waiting for app: TestDataTypes to get deployed
2021/12/16 11:35:23 Waiting for app: TestDataTypes to get deployed
2021/12/16 11:35:28 Waiting for app: TestDataTypes to get deployed
2021/12/16 11:35:28 App: TestDataTypes got deployed
2021/12/16 11:35:28 Starting bucket ops to source bucket
2021/12/16 11:35:33 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/16 11:35:38 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/16 11:35:43 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/16 11:35:43 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/16 11:35:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:35:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:35:48 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 11:35:53 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 11:35:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 11:36:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 11:36:03 App: TestDataTypes got un-deployed
2021/12/16 11:36:03 No eventing-con process running
2021/12/16 11:36:04 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/16 11:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:36:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 11:36:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:36:27 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/16 11:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.08s)
=== RUN   TestEnoentGet
2021/12/16 11:36:27 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:36:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/16 11:36:33 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:36:38 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:36:43 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:36:48 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:36:48 App: TestEnoentGet got deployed
2021/12/16 11:36:48 Starting bucket ops to source bucket
2021/12/16 11:36:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:36:59 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 11:36:59 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:36:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:36:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:37:04 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:37:09 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:37:14 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:37:19 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:37:19 App: TestEnoentGet got un-deployed
2021/12/16 11:37:19 No eventing-con process running
2021/12/16 11:37:19 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/16 11:37:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:37:30 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/16 11:37:35 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/16 11:37:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:37:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:37:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:37:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:37:47 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:37:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/16 11:37:53 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:37:58 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:38:03 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:38:08 Waiting for app: TestEnoentGet to get deployed
2021/12/16 11:38:08 App: TestEnoentGet got deployed
2021/12/16 11:38:08 Starting bucket ops to source bucket
2021/12/16 11:38:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:38:19 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 11:38:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:38:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:38:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:38:24 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:38:29 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:38:34 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:38:39 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 11:38:39 App: TestEnoentGet got un-deployed
2021/12/16 11:38:39 No eventing-con process running
2021/12/16 11:38:39 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/16 11:38:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:38:51 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/16 11:38:56 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/16 11:38:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:38:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:39:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:39:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.22s)
=== RUN   TestEnoentDelete
2021/12/16 11:39:07 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:39:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 11:39:13 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:39:18 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:39:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:39:29 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:39:29 App: TestEnoentDelete got deployed
2021/12/16 11:39:29 Starting bucket ops to source bucket
2021/12/16 11:39:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:39:39 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 11:39:39 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:39:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:39:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:39:44 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:39:49 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:39:54 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:39:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:39:59 App: TestEnoentDelete got un-deployed
2021/12/16 11:39:59 No eventing-con process running
2021/12/16 11:39:59 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/16 11:40:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:40:11 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/16 11:40:16 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/16 11:40:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:40:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:40:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:40:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:40:28 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:40:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 11:40:34 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:40:39 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:40:44 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:40:49 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 11:40:49 App: TestEnoentDelete got deployed
2021/12/16 11:40:49 Starting bucket ops to source bucket
2021/12/16 11:40:54 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:40:59 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 11:40:59 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:40:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:40:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:41:04 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:41:09 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:41:14 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:41:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 11:41:19 App: TestEnoentDelete got un-deployed
2021/12/16 11:41:19 No eventing-con process running
2021/12/16 11:41:19 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/16 11:41:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:41:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:41:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:41:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:41:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:41:48 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/16 11:41:53 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/16 11:41:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (165.48s)
=== RUN   TestError
2021/12/16 11:42:03 Waiting for app: TestError to get un-deployed
2021/12/16 11:42:03 App: TestError got un-deployed
2021/12/16 11:42:03 No eventing-con process running
2021/12/16 11:42:03 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/16 11:42:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:42:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:42:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:42:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:42:16 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:42:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/16 11:42:23 Waiting for app: TestError to get deployed
2021/12/16 11:42:28 Waiting for app: TestError to get deployed
2021/12/16 11:42:33 Waiting for app: TestError to get deployed
2021/12/16 11:42:38 Waiting for app: TestError to get deployed
2021/12/16 11:42:38 App: TestError got deployed
2021/12/16 11:42:38 Starting bucket ops to source bucket
2021/12/16 11:42:43 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/16 11:42:48 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/16 11:42:53 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/16 11:42:53 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/16 11:42:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:42:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:42:58 Waiting for app: TestError to get un-deployed
2021/12/16 11:43:03 Waiting for app: TestError to get un-deployed
2021/12/16 11:43:08 Waiting for app: TestError to get un-deployed
2021/12/16 11:43:13 Waiting for app: TestError to get un-deployed
2021/12/16 11:43:13 App: TestError got un-deployed
2021/12/16 11:43:13 No eventing-con process running
2021/12/16 11:43:13 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/16 11:43:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:43:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:43:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:43:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:43:37 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/16 11:43:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.78s)
=== RUN   TestCRLF
2021/12/16 11:43:47 Waiting for app: TestCRLF to get un-deployed
2021/12/16 11:43:47 App: TestCRLF got un-deployed
2021/12/16 11:43:47 No eventing-con process running
2021/12/16 11:43:47 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/16 11:43:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:43:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:44:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:44:00 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:44:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/16 11:44:06 Waiting for app: TestCRLF to get deployed
2021/12/16 11:44:11 Waiting for app: TestCRLF to get deployed
2021/12/16 11:44:16 Waiting for app: TestCRLF to get deployed
2021/12/16 11:44:21 Waiting for app: TestCRLF to get deployed
2021/12/16 11:44:21 App: TestCRLF got deployed
2021/12/16 11:44:21 Starting bucket ops to source bucket
2021/12/16 11:44:27 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:44:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 11:44:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 11:44:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:44:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:44:37 Waiting for app: TestCRLF to get un-deployed
2021/12/16 11:44:42 Waiting for app: TestCRLF to get un-deployed
2021/12/16 11:44:47 Waiting for app: TestCRLF to get un-deployed
2021/12/16 11:44:52 Waiting for app: TestCRLF to get un-deployed
2021/12/16 11:44:52 App: TestCRLF got un-deployed
2021/12/16 11:44:52 No eventing-con process running
2021/12/16 11:44:52 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/16 11:44:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:45:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 11:45:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 11:45:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:45:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:45:20 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/16 11:45:25 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/16 11:45:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.85s)
=== RUN   TestImportExport
2021/12/16 11:45:35 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 11:45:35 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 11:45:35 No eventing-con process running
2021/12/16 11:45:35 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/16 11:45:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:45:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:45:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:45:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:45:54 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 11:45:54 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 11:45:54 No eventing-con process running
2021/12/16 11:45:54 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/16 11:45:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:46:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:46:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:46:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:46:08 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:46:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 11:46:09 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:46:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 11:46:15 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 11:46:20 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 11:46:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 11:46:30 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 11:46:30 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/16 11:46:35 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 11:46:40 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 11:46:45 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 11:46:45 App: TestImportExport_bucket_op_on_update got deployed
2021/12/16 11:46:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:46:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 11:46:55 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 11:47:00 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 11:47:05 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 11:47:05 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 11:47:05 No eventing-con process running
2021/12/16 11:47:05 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/16 11:47:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:47:10 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 11:47:15 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 11:47:20 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 11:47:25 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 11:47:25 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 11:47:25 No eventing-con process running
2021/12/16 11:47:26 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/16 11:47:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 11:47:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 11:47:43 No eventing-con process running
2021/12/16 11:47:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/16 11:47:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:47:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:47:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:48:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 11:48:01 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 11:48:01 No eventing-con process running
2021/12/16 11:48:01 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/16 11:48:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:48:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:48:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:48:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.70s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/16 11:48:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:48:25 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 11:48:25 No eventing-con process running
2021/12/16 11:48:25 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/16 11:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:48:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:48:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:48:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:48:39 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:48:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 11:48:40 Starting bucket ops to source bucket
2021/12/16 11:48:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:48:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:48:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:49:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:49:08 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 11:49:08 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:49:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 11:49:13 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 11:49:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:49:13 Undeploying app: bucket_op_on_update
2021/12/16 11:49:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:49:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:49:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:49:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:49:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:49:33 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 11:49:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:49:33 No eventing-con process running
2021/12/16 11:49:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:49:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:50:17 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:50:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 11:50:18 Starting bucket ops to source bucket
2021/12/16 11:50:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:50:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:50:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:50:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:50:46 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 11:50:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:50:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 11:50:51 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 11:50:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:50:51 Undeploying app: bucket_op_on_update
2021/12/16 11:50:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:50:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:51:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:51:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:51:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:51:12 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 11:51:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:51:12 No eventing-con process running
2021/12/16 11:51:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:51:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:51:56 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:51:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 11:51:57 Starting bucket ops to source bucket
2021/12/16 11:52:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:52:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:52:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:52:20 Waiting for dst bucket item count to get to: 5000 curr count: 1314
2021/12/16 11:52:25 Waiting for dst bucket item count to get to: 5000 curr count: 1314
2021/12/16 11:52:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:52:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 11:52:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 11:52:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:52:30 Undeploying app: bucket_op_on_update
2021/12/16 11:52:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:52:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:52:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:52:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:52:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:52:50 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 11:52:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:52:50 No eventing-con process running
2021/12/16 11:52:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:52:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:53:34 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:53:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 11:53:35 Starting bucket ops to source bucket
2021/12/16 11:53:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:53:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:53:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:53:58 Waiting for dst bucket item count to get to: 5000 curr count: 3026
2021/12/16 11:54:03 Waiting for dst bucket item count to get to: 5000 curr count: 3026
2021/12/16 11:54:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:54:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 11:54:08 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 11:54:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:54:08 Undeploying app: bucket_op_on_update
2021/12/16 11:54:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:54:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:54:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:54:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:54:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:54:28 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 11:54:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:54:28 No eventing-con process running
2021/12/16 11:54:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:54:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:55:12 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:55:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 11:55:13 Starting bucket ops to source bucket
2021/12/16 11:55:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:55:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:55:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:55:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 11:55:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:55:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 11:55:41 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 11:55:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:55:41 Undeploying app: bucket_op_on_update
2021/12/16 11:55:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:55:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:55:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:55:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:56:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 11:56:02 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 11:56:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:56:02 No eventing-con process running
2021/12/16 11:56:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:56:46 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.70s)
=== RUN   TestOnUpdateN1QLOp
2021/12/16 11:56:56 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 11:56:56 App: TestOnUpdateN1QLOp got un-deployed
2021/12/16 11:56:56 No eventing-con process running
2021/12/16 11:56:56 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/16 11:56:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:57:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:57:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:57:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:57:09 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:57:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/16 11:57:11 Starting bucket ops to source bucket
2021/12/16 11:57:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:57:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:57:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:57:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:57:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:57:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:57:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:57:39 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 11:57:44 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 11:57:49 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 11:57:54 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 11:57:54 App: TestOnUpdateN1QLOp got un-deployed
2021/12/16 11:57:54 No eventing-con process running
2021/12/16 11:57:54 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/16 11:57:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:58:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 11:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:58:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:58:17 Waiting for dst bucket item count to get to: 0 curr count: 2468
2021/12/16 11:58:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (91.64s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/16 11:58:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 11:58:27 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/16 11:58:27 No eventing-con process running
2021/12/16 11:58:27 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/16 11:58:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:58:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:58:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:58:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:58:41 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 11:58:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 11:58:42 Starting bucket ops to source bucket
2021/12/16 11:58:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:58:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:59:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:59:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 11:59:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 11:59:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 11:59:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 11:59:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 11:59:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 11:59:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 11:59:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 11:59:25 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/16 11:59:25 No eventing-con process running
2021/12/16 11:59:25 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/16 11:59:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:59:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 11:59:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 11:59:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 11:59:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 11:59:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 11:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.50s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/16 12:00:04 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 12:00:04 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 12:00:04 No eventing-con process running
2021/12/16 12:00:04 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/16 12:00:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:00:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:00:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:00:18 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:00:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 12:00:19 Starting bucket ops to source bucket
2021/12/16 12:00:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:00:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:00:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:00:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:00:47 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 12:00:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:00:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:00:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:00:52 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 12:00:57 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 12:01:02 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 12:01:07 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 12:01:07 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 12:01:07 No eventing-con process running
2021/12/16 12:01:07 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/16 12:01:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:01:19 Waiting for dst bucket item count to get to: 0 curr count: 4827
2021/12/16 12:01:24 Waiting for dst bucket item count to get to: 0 curr count: 2327
2021/12/16 12:01:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:01:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:01:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:01:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.63s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/16 12:01:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 12:01:46 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/16 12:01:46 No eventing-con process running
2021/12/16 12:01:46 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/16 12:01:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:01:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:01:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:01:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:01:59 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:02:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/16 12:02:00 Starting bucket ops to source bucket
2021/12/16 12:02:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:02:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:02:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:02:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:02:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:02:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:02:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:02:29 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 12:02:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 12:02:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 12:02:44 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 12:02:44 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/16 12:02:44 No eventing-con process running
2021/12/16 12:02:44 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/16 12:02:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:02:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:02:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:02:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:03:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:03:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (91.69s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/16 12:03:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 12:03:17 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/16 12:03:17 No eventing-con process running
2021/12/16 12:03:17 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/16 12:03:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:03:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:03:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:03:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:03:31 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:03:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/16 12:03:33 Starting bucket ops to source bucket
2021/12/16 12:03:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:03:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:03:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:03:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:03:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:03:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:03:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:04:01 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 12:04:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 12:04:11 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 12:04:16 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 12:04:16 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/16 12:04:16 No eventing-con process running
2021/12/16 12:04:16 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/16 12:04:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:04:28 Waiting for dst bucket item count to get to: 0 curr count: 2818
2021/12/16 12:04:33 Waiting for dst bucket item count to get to: 0 curr count: 318
2021/12/16 12:04:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:04:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:04:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.21s)
=== RUN   TestOnDeleteBucketOp
2021/12/16 12:04:55 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 12:04:55 App: TestOnDeleteBucketOp got un-deployed
2021/12/16 12:04:55 No eventing-con process running
2021/12/16 12:04:55 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/16 12:04:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:05:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:05:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:05:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:05:08 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:05:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 12:05:09 Starting bucket ops to source bucket
2021/12/16 12:05:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:05:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:05:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:05:35 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/16 12:05:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:05:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:05:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:05:40 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 12:05:45 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 12:05:50 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 12:05:55 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 12:05:55 App: TestOnDeleteBucketOp got un-deployed
2021/12/16 12:05:55 No eventing-con process running
2021/12/16 12:05:55 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/16 12:05:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:06:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:06:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:06:14 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:06:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.93s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/16 12:06:24 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 12:06:24 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/16 12:06:24 No eventing-con process running
2021/12/16 12:06:24 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/16 12:06:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:06:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:06:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:06:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:06:38 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:06:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/16 12:06:39 Starting bucket ops to source bucket
2021/12/16 12:06:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:06:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:07:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:07:05 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/16 12:07:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:07:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:07:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:07:10 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 12:07:15 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 12:07:20 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 12:07:25 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 12:07:25 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/16 12:07:25 No eventing-con process running
2021/12/16 12:07:25 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/16 12:07:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:07:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:07:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:07:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:07:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.93s)
=== RUN   TestTimerBucketOp
2021/12/16 12:07:54 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 12:07:54 App: TestTimerBucketOp got un-deployed
2021/12/16 12:07:54 No eventing-con process running
2021/12/16 12:07:54 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/16 12:07:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:08:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:08:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:08:08 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:08:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 12:08:09 Starting bucket ops to source bucket
2021/12/16 12:08:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:08:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:08:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:08:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:08:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:08:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:08:47 Waiting for dst bucket item count to get to: 5000 curr count: 1716
2021/12/16 12:08:52 Waiting for dst bucket item count to get to: 5000 curr count: 4216
2021/12/16 12:08:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:08:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:08:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:08:57 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 12:09:02 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 12:09:07 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 12:09:12 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 12:09:12 App: TestTimerBucketOp got un-deployed
2021/12/16 12:09:12 No eventing-con process running
2021/12/16 12:09:12 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/16 12:09:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:09:24 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:09:29 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 12:09:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:09:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:09:41 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:09:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:09:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.70s)
=== RUN   TestTimerInPastBucketOp
2021/12/16 12:09:56 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 12:09:56 App: TestTimerInPastBucketOp got un-deployed
2021/12/16 12:09:56 No eventing-con process running
2021/12/16 12:09:56 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/16 12:09:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:10:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:10:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:10:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:10:10 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:10:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 12:10:11 Starting bucket ops to source bucket
2021/12/16 12:10:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:10:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:10:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:10:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:10:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:10:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:10:49 Waiting for dst bucket item count to get to: 5000 curr count: 2135
2021/12/16 12:10:54 Waiting for dst bucket item count to get to: 5000 curr count: 2135
2021/12/16 12:10:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:10:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:10:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:10:59 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 12:11:04 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 12:11:09 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 12:11:14 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 12:11:14 App: TestTimerInPastBucketOp got un-deployed
2021/12/16 12:11:14 No eventing-con process running
2021/12/16 12:11:14 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/16 12:11:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:11:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:11:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:11:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:11:38 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 12:11:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.56s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/16 12:11:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:11:48 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 12:11:48 No eventing-con process running
2021/12/16 12:11:48 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/16 12:11:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:11:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:11:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:12:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:12:01 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:12:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 12:12:02 Starting bucket ops to source bucket
2021/12/16 12:12:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:12:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:12:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:12:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:12:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:12:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:12:40 Waiting for dst bucket item count to get to: 5000 curr count: 734
2021/12/16 12:12:45 Waiting for dst bucket item count to get to: 5000 curr count: 734
2021/12/16 12:12:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:12:50 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 12:12:50 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 12:12:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:12:50 Undeploying app: bucket_op_with_timer
2021/12/16 12:12:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:12:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:13:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:13:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:13:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:13:10 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 12:13:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:13:10 No eventing-con process running
2021/12/16 12:13:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:13:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:13:54 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:13:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 12:13:56 Starting bucket ops to source bucket
2021/12/16 12:14:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:14:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:14:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:14:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:14:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:14:29 Waiting for dst bucket item count to get to: 5000 curr count: 2751
2021/12/16 12:14:34 Waiting for dst bucket item count to get to: 5000 curr count: 2751
2021/12/16 12:14:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:14:39 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 12:14:39 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 12:14:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:14:39 Undeploying app: bucket_op_with_timer
2021/12/16 12:14:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:14:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:14:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:14:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:14:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:14:59 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 12:14:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:14:59 No eventing-con process running
2021/12/16 12:15:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:15:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:15:43 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:15:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 12:15:44 Starting bucket ops to source bucket
2021/12/16 12:15:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:15:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:16:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:16:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:16:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:16:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:16:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:16:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 12:16:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:16:32 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 12:16:32 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 12:16:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:16:32 Undeploying app: bucket_op_with_timer
2021/12/16 12:16:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:16:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:16:42 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:16:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:16:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:16:52 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 12:16:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:16:52 No eventing-con process running
2021/12/16 12:16:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:17:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:17:36 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:17:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 12:17:37 Starting bucket ops to source bucket
2021/12/16 12:17:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:17:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:17:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:18:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:18:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:18:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:18:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:18:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:18:20 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 12:18:20 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 12:18:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:18:20 Undeploying app: bucket_op_with_timer
2021/12/16 12:18:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:18:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:18:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:18:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:18:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:18:40 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 12:18:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:18:40 No eventing-con process running
2021/12/16 12:18:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:18:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:19:24 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:19:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 12:19:26 Starting bucket ops to source bucket
2021/12/16 12:19:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:19:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:19:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:19:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:19:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:19:59 Waiting for dst bucket item count to get to: 5000 curr count: 603
2021/12/16 12:20:04 Waiting for dst bucket item count to get to: 5000 curr count: 603
2021/12/16 12:20:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:20:09 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 12:20:09 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 12:20:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:20:09 Undeploying app: bucket_op_with_timer
2021/12/16 12:20:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:20:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:20:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:20:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:20:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 12:20:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 12:20:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:20:29 No eventing-con process running
2021/12/16 12:20:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:20:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:21:13 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (575.19s)
=== RUN   TestMultipleHandlers
2021/12/16 12:21:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 12:21:23 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/16 12:21:23 No eventing-con process running
2021/12/16 12:21:23 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/16 12:21:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:21:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:21:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:21:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:21:41 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 12:21:41 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/16 12:21:41 No eventing-con process running
2021/12/16 12:21:41 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/16 12:21:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:21:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:21:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:21:55 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:21:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 12:21:56 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:21:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/16 12:22:02 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 12:22:07 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 12:22:12 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 12:22:17 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 12:22:17 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/16 12:22:22 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 12:22:27 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 12:22:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 12:22:32 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/16 12:22:32 Starting bucket ops to source bucket
2021/12/16 12:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:22:48 Waiting for dst bucket item count to get to: 5000 curr count: 4855
2021/12/16 12:22:53 Waiting for dst bucket item count to get to: 5000 curr count: 4855
2021/12/16 12:22:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:22:53 Starting bucket ops to source bucket
2021/12/16 12:23:08 Waiting for dst bucket item count to get to: 10000 curr count: 5892
2021/12/16 12:23:13 Waiting for dst bucket item count to get to: 10000 curr count: 5892
2021/12/16 12:23:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 12:23:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:23:13 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:23:13 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:23:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:23:18 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 12:23:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 12:23:28 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 12:23:33 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 12:23:33 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/16 12:23:33 No eventing-con process running
2021/12/16 12:23:33 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/16 12:23:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:23:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:23:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:23:52 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 12:23:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 12:23:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:23:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:24:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 12:24:07 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 12:24:12 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 12:24:17 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 12:24:17 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/16 12:24:17 No eventing-con process running
2021/12/16 12:24:17 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/16 12:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:24:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:24:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:24:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.18s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/16 12:24:41 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 12:24:41 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/16 12:24:41 No eventing-con process running
2021/12/16 12:24:41 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/16 12:24:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:24:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:24:55 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:24:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 12:24:56 Starting bucket ops to source bucket
2021/12/16 12:25:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:25:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:25:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:25:19 Waiting for dst bucket item count to get to: 5000 curr count: 3014
2021/12/16 12:25:24 Waiting for dst bucket item count to get to: 5000 curr count: 3014
2021/12/16 12:25:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:25:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:25:29 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 12:25:29 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 12:25:29 Pausing app: bucket_op_on_update
2021/12/16 12:25:29 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:25:34 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 12:25:34 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 12:25:34 Resuming app: bucket_op_on_update
2021/12/16 12:25:35 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:25:35 Starting bucket ops to source bucket
2021/12/16 12:25:43 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:25:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:25:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:25:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 12:25:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:25:58 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 12:25:58 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 12:25:58 Pausing app: bucket_op_on_update
2021/12/16 12:25:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:26:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 12:26:03 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 12:26:03 Resuming app: bucket_op_on_update
2021/12/16 12:26:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:26:04 Starting bucket ops to source bucket
2021/12/16 12:26:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:26:17 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:26:22 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:26:22 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/16 12:26:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:26:27 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 12:26:27 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 12:26:27 Pausing app: bucket_op_on_update
2021/12/16 12:26:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:26:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 12:26:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 12:26:32 Resuming app: bucket_op_on_update
2021/12/16 12:26:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:26:34 Starting bucket ops to source bucket
2021/12/16 12:26:42 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:26:47 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:26:52 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:26:52 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 12:26:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:26:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 12:26:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 12:26:57 Pausing app: bucket_op_on_update
2021/12/16 12:26:57 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:27:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 12:27:02 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 12:27:02 Resuming app: bucket_op_on_update
2021/12/16 12:27:03 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:27:03 Starting bucket ops to source bucket
2021/12/16 12:27:11 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:27:16 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:27:21 Waiting for dst bucket item count to get to: 25000 curr count: 20426
2021/12/16 12:27:26 Waiting for dst bucket item count to get to: 25000 curr count: 20426
2021/12/16 12:27:26 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 12:27:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:27:31 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 12:27:31 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 12:27:31 Pausing app: bucket_op_on_update
2021/12/16 12:27:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:27:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 12:27:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 12:27:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:27:41 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 12:27:46 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 12:27:51 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 12:27:56 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 12:27:56 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/16 12:27:56 No eventing-con process running
2021/12/16 12:27:56 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/16 12:27:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:28:08 Waiting for dst bucket item count to get to: 0 curr count: 17010
2021/12/16 12:28:13 Waiting for dst bucket item count to get to: 0 curr count: 4510
2021/12/16 12:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:28:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:28:25 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 12:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.95s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/16 12:28:35 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 12:28:35 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/16 12:28:35 No eventing-con process running
2021/12/16 12:28:35 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/16 12:28:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:28:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:28:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:28:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:28:49 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:28:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 12:28:50 Starting bucket ops to source bucket
2021/12/16 12:28:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:29:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:29:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:29:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:29:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:29:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:29:28 Waiting for dst bucket item count to get to: 5000 curr count: 1539
2021/12/16 12:29:33 Waiting for dst bucket item count to get to: 5000 curr count: 4039
2021/12/16 12:29:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:29:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:29:38 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 12:29:38 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 12:29:38 Pausing app: bucket_op_with_timer
2021/12/16 12:29:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:29:43 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 12:29:43 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 12:29:43 Resuming app: bucket_op_with_timer
2021/12/16 12:29:44 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:29:44 Starting bucket ops to source bucket
2021/12/16 12:29:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:29:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:30:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:30:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:30:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:30:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 12:30:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:30:17 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 12:30:17 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 12:30:17 Pausing app: bucket_op_with_timer
2021/12/16 12:30:17 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:30:22 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 12:30:22 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 12:30:22 Resuming app: bucket_op_with_timer
2021/12/16 12:30:23 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:30:23 Starting bucket ops to source bucket
2021/12/16 12:30:31 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:30:36 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:30:41 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:30:46 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:30:51 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 12:30:51 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/16 12:30:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:30:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 12:30:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 12:30:56 Pausing app: bucket_op_with_timer
2021/12/16 12:30:57 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:31:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 12:31:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 12:31:02 Resuming app: bucket_op_with_timer
2021/12/16 12:31:03 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:31:03 Starting bucket ops to source bucket
2021/12/16 12:31:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:31:16 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:31:21 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:31:26 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 12:31:31 Waiting for dst bucket item count to get to: 20000 curr count: 16621
2021/12/16 12:31:36 Waiting for dst bucket item count to get to: 20000 curr count: 16621
2021/12/16 12:31:36 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 12:31:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:31:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 12:31:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 12:31:41 Pausing app: bucket_op_with_timer
2021/12/16 12:31:41 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:31:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 12:31:46 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 12:31:46 Resuming app: bucket_op_with_timer
2021/12/16 12:31:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:31:47 Starting bucket ops to source bucket
2021/12/16 12:31:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:32:00 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:32:05 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:32:10 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:32:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 12:32:20 Waiting for dst bucket item count to get to: 25000 curr count: 21243
2021/12/16 12:32:25 Waiting for dst bucket item count to get to: 25000 curr count: 21243
2021/12/16 12:32:25 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 12:32:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:32:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 12:32:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 12:32:30 Pausing app: bucket_op_with_timer
2021/12/16 12:32:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:32:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 12:32:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 12:32:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:32:40 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 12:32:45 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 12:32:50 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 12:32:55 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 12:32:55 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/16 12:32:55 No eventing-con process running
2021/12/16 12:32:56 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/16 12:32:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:33:07 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 12:33:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:33:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:33:19 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 12:33:24 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/16 12:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (299.25s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/16 12:33:34 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 12:33:34 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/16 12:33:34 No eventing-con process running
2021/12/16 12:33:34 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/16 12:33:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:33:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:33:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:33:48 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:33:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/16 12:33:54 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 12:33:59 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 12:34:04 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 12:34:09 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 12:34:09 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/16 12:34:09 Starting bucket ops to source bucket
2021/12/16 12:34:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:34:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:34:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:34:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:34:38 Waiting for dst bucket item count to get to: 5000 curr count: 2031
2021/12/16 12:34:43 Waiting for dst bucket item count to get to: 5000 curr count: 4182
2021/12/16 12:34:48 Waiting for dst bucket item count to get to: 5000 curr count: 4651
2021/12/16 12:34:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:34:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:34:48 Pausing app: bucket_op_with_timer
2021/12/16 12:34:49 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:34:54 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/16 12:34:54 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/16 12:34:54 Resuming app: bucket_op_with_timer
2021/12/16 12:34:55 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:34:55 Starting bucket ops to source bucket
2021/12/16 12:35:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:35:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:35:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:35:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:35:26 Waiting for dst bucket item count to get to: 10000 curr count: 5749
2021/12/16 12:35:31 Waiting for dst bucket item count to get to: 10000 curr count: 7866
2021/12/16 12:35:31 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 12:35:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:35:37 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 12:35:42 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 12:35:47 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 12:35:52 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 12:35:52 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/16 12:35:52 No eventing-con process running
2021/12/16 12:35:52 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/16 12:35:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:36:03 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 12:36:08 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:36:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:36:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:36:20 Waiting for dst bucket item count to get to: 0 curr count: 7505
2021/12/16 12:36:25 Waiting for dst bucket item count to get to: 0 curr count: 7505
2021/12/16 12:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.06s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/16 12:36:35 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 12:36:35 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/16 12:36:35 No eventing-con process running
2021/12/16 12:36:35 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/16 12:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:36:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:36:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:36:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:36:49 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:36:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 12:36:55 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 12:37:00 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 12:37:05 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 12:37:10 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 12:37:10 App: TestPauseResumeWithEventingReb got deployed
2021/12/16 12:37:10 Starting bucket ops to source bucket
2021/12/16 12:37:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:37:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:37:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:37:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:37:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:37:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:37:34 Pausing app: bucket_op_with_timer
2021/12/16 12:37:34 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:37:39 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/16 12:37:39 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/16 12:37:39 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 12:37:40 Removing node(s): [] from the cluster
2021/12/16 12:37:45 Rebalance progress: 29.16666666666666
2021/12/16 12:37:50 Rebalance progress: 29.16666666666666
2021/12/16 12:37:55 Rebalance progress: 100
2021/12/16 12:37:55 Resuming app: bucket_op_with_timer
2021/12/16 12:37:56 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:37:56 Starting bucket ops to source bucket
2021/12/16 12:38:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:38:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:38:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:38:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 12:38:28 Waiting for dst bucket item count to get to: 10000 curr count: 6471
2021/12/16 12:38:33 Waiting for dst bucket item count to get to: 10000 curr count: 6471
2021/12/16 12:38:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 12:38:33 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 12:38:38 Rebalance progress: 29.16666666666666
2021/12/16 12:38:43 Rebalance progress: 29.16666666666666
2021/12/16 12:38:48 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/16 12:38:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:39:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:39:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:39:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:39:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:40:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:40:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:40:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:40:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:41:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:41:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:41:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 12:41:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: fcb202d91a8ed4ccd606f6d0b5ed4337
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
Producer node: 172.31.2.134:9300	Node UUID: 1008c29742d57a2334ce9ce19d7f63b3
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
Producer node: 127.0.0.1:9303	Node UUID: 14baf2a7735e0ceb7e39eaeb73063ef9
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]


2021/12/16 12:42:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:42:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:42:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 12:42:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 12:42:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 12:42:29 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 12:42:29 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/16 12:42:29 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:42:29 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/16 12:42:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:42:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8133
2021/12/16 12:42:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8133
2021/12/16 12:42:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:42:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:42:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 12:42:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (392.06s)
=== RUN   TestBucketDeleteAfterPause
2021/12/16 12:43:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/16 12:43:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/16 12:43:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:43:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/16 12:43:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:43:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:43:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:43:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:43:21 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:43:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/16 12:43:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 12:43:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 12:43:37 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 12:43:42 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 12:43:42 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/16 12:43:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:43:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:43:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:43:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:43:52 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/16 12:43:52 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:43:57 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/16 12:43:57 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/16 12:43:57 handler_test.go:716: Deleting source bucket: default
2021/12/16 12:44:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/16 12:44:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/16 12:44:08 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/16 12:44:08 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/16 12:44:08 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:44:08 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/16 12:44:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:44:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:44:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:44:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:44:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (88.96s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/16 12:44:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 12:44:36 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/16 12:44:36 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:44:36 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/16 12:44:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:44:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:44:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:44:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:44:49 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:44:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/16 12:44:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 12:45:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 12:45:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 12:45:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 12:45:11 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/16 12:45:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:45:46 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/16 12:45:46 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:45:51 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/16 12:45:51 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/16 12:48:51 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/16 12:48:51 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/16 12:48:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/16 12:48:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 12:48:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:48:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/16 12:49:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 12:49:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:49:02 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3061 expected 5000
2021/12/16 12:49:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 12:49:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:49:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/16 12:49:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 12:49:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:49:12 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/16 12:49:12 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/16 12:49:12 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:49:22 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 12:49:27 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 12:49:32 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 12:49:32 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/16 12:49:32 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:49:32 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/16 12:49:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:49:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:49:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 12:49:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:49:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:49:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.90s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/16 12:50:05 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 12:50:05 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/16 12:50:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:50:05 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/16 12:50:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:50:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:50:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:50:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:50:18 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:50:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 12:50:24 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:50:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:50:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:50:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:50:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/16 12:50:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:50:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:50:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:50:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 12:50:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:50:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:50:59 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/16 12:50:59 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:51:29 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/16 12:51:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 12:51:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:51:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:51:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:51:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 12:51:50 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/16 12:51:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:52:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:52:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 345
2021/12/16 12:52:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 345
2021/12/16 12:52:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:52:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:52:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/16 12:52:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:52:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 12:52:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 12:52:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 12:52:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/16 12:52:33 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:52:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/16 12:52:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:52:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:52:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:52:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (170.74s)
=== RUN   TestCPPWorkerCleanup
2021/12/16 12:52:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 12:52:56 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/16 12:52:56 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:52:56 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/16 12:52:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:53:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:53:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:53:09 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:53:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/16 12:53:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:53:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:53:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:53:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:53:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:53:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4662
2021/12/16 12:53:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4662
2021/12/16 12:53:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:53:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:53:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:53:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 12:53:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 12:53:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 12:54:03 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 12:54:03 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/16 12:54:03 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:54:03 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/16 12:54:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:54:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:54:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 12:54:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:54:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:54:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/16 12:54:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4452
2021/12/16 12:54:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.81s)
=== RUN   TestWithUserXattrs
2021/12/16 12:55:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 12:55:16 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/16 12:55:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:55:16 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/16 12:55:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:55:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:55:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:55:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:55:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:55:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:55:33 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:55:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/16 12:55:39 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 12:55:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 12:55:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 12:55:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 12:55:54 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/16 12:55:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:56:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:56:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4209
2021/12/16 12:56:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4209
2021/12/16 12:56:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:56:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:56:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 12:56:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 12:56:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 12:56:29 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 12:56:34 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 12:56:34 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/16 12:56:34 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:56:34 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/16 12:56:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:56:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:56:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:56:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:56:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:56:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.01s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/16 12:57:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 12:57:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 12:57:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:57:07 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/16 12:57:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:57:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:57:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:57:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:57:20 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/16 12:57:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/16 12:57:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 12:57:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 12:57:36 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 12:57:41 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 12:57:41 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/16 12:57:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:57:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:57:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 12:57:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 12:57:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:57:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:58:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3211
2021/12/16 12:58:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 711
2021/12/16 12:58:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:58:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:58:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 12:58:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:58:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 12:58:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14858
2021/12/16 12:58:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14858
2021/12/16 12:58:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 12:58:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 12:58:56 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 12:59:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 12:59:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 12:59:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 12:59:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 12:59:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 12:59:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:59:22 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 12:59:22 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 12:59:22 eventingSetup.go:46: No eventing-con process running
2021/12/16 12:59:22 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/16 12:59:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:59:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 12:59:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/16 12:59:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 12:59:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 12:59:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/16 12:59:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/16 12:59:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (178.02s)
=== RUN   TestMetaBucketDelete
2021/12/16 13:00:05 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 13:00:05 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 13:00:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:00:05 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/16 13:00:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:00:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:00:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:00:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:00:18 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:00:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 13:00:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 13:00:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 13:00:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 13:00:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 13:00:39 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/16 13:00:39 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/16 13:00:40 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/16 13:00:45 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 13:00:45 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 13:01:00 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 13:01:00 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 13:01:00 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:01:01 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/16 13:01:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:01:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:01:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:01:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.87s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/16 13:01:24 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:01:24 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:01:24 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:01:24 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 13:01:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:01:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:01:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:01:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:01:37 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:01:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 13:01:57 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/16 13:01:58 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/16 13:02:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:02:13 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:02:28 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:02:28 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:02:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:02:28 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 13:02:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:02:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:02:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:02:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.65s)
=== RUN   TestSourceBucketDelete
2021/12/16 13:02:51 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 13:02:51 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 13:02:51 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:02:51 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/16 13:02:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:02:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:02:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:03:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:03:04 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:03:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 13:03:11 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 13:03:16 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 13:03:21 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 13:03:26 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 13:03:26 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/16 13:03:26 handler_test.go:945: Deleting source bucket: default
2021/12/16 13:03:27 handler_test.go:947: Deleted source bucket: default
2021/12/16 13:03:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 13:03:32 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 13:03:47 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 13:03:47 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 13:03:47 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:03:47 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/16 13:03:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:03:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:03:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:04:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.80s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/16 13:04:10 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:04:10 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:04:10 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:04:10 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 13:04:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:04:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:04:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:04:23 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:04:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 13:04:43 handler_test.go:966: Deleting source bucket: default
2021/12/16 13:04:45 handler_test.go:968: Deleted source bucket: default
2021/12/16 13:05:00 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:05:00 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:05:15 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:05:15 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:05:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:05:15 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 13:05:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:05:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:05:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:05:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.79s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/16 13:05:38 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:05:38 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:05:38 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:05:38 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 13:05:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:05:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:05:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:05:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:05:51 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:05:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 13:06:11 handler_test.go:988: Deleting source bucket: default
2021/12/16 13:06:12 handler_test.go:990: Deleted source bucket: default
2021/12/16 13:06:12 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/16 13:06:13 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/16 13:06:28 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:06:28 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:06:43 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 13:06:43 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 13:06:43 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:06:43 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 13:06:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:06:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:06:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:06:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.14s)
=== RUN   TestUndeployDuringBootstrap
2021/12/16 13:07:06 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 13:07:06 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 13:07:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:07:06 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/16 13:07:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:07:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:07:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:07:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:07:19 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:07:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 13:07:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 13:07:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:07:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:07:33 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 13:07:33 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/16 13:07:33 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 13:07:38 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/16 13:07:43 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 13:07:48 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 13:07:53 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 13:07:53 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 13:07:53 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:08:18 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 13:08:18 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 13:08:18 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:08:18 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/16 13:08:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:08:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 13:08:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 13:08:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:08:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:08:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3240
2021/12/16 13:08:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.76s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/16 13:08:51 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:08:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/16 13:08:57 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 13:09:02 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 13:09:07 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 13:09:12 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 13:09:12 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/16 13:09:12 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/16 13:09:12 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 13:09:12 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/16 13:09:17 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/16 13:09:17 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/16 13:09:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:09:17 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/16 13:09:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:09:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:09:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:09:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.67s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/16 13:09:40 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 13:09:40 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 13:09:40 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:09:40 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/16 13:09:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:09:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:09:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:09:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:09:54 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:09:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/16 13:10:00 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 13:10:05 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 13:10:10 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 13:10:15 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 13:10:15 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/16 13:10:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 13:10:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:10:50 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 13:10:55 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 13:11:00 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 13:11:05 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 13:11:05 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 13:11:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:12:50 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 13:12:50 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 13:12:50 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:12:50 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/16 13:12:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:13:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 13:13:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 13:13:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:13:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:13:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4265
2021/12/16 13:13:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1765
2021/12/16 13:13:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (233.02s)
=== RUN   TestUndeployWithKVFailover
2021/12/16 13:13:33 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 13:13:33 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/16 13:13:33 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:13:33 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/16 13:13:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:13:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:13:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:13:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:13:47 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:13:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/16 13:13:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 13:13:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:14:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:14:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:14:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:14:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:14:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 443
2021/12/16 13:14:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 443
2021/12/16 13:14:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 13:14:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 13:14:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 13:14:31 Rebalance progress: 29.16666666666666
2021/12/16 13:14:36 Rebalance progress: 29.16666666666666
2021/12/16 13:14:41 Rebalance progress: 100
2021/12/16 13:14:41 Failing over: [127.0.0.1:19003]
2021/12/16 13:14:51 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 13:15:51 Removing node(s): [] from the cluster
2021/12/16 13:15:56 Rebalance progress: 29.16666666666666
2021/12/16 13:16:01 Rebalance progress: 29.16666666666666
2021/12/16 13:16:06 Rebalance progress: 100
2021/12/16 13:16:11 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 13:16:11 App: TestUndeployWithKVFailover got un-deployed
2021/12/16 13:16:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:16:16 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 13:16:16 App: TestUndeployWithKVFailover got un-deployed
2021/12/16 13:16:16 No eventing-con process running
2021/12/16 13:16:16 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/16 13:16:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:16:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 13:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:16:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:16:39 Waiting for dst bucket item count to get to: 0 curr count: 4969
2021/12/16 13:16:44 Waiting for dst bucket item count to get to: 0 curr count: 2469
2021/12/16 13:16:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.51s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/16 13:16:54 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 13:16:54 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/16 13:16:54 No eventing-con process running
2021/12/16 13:16:54 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/16 13:16:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:17:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:17:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:17:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:17:07 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:17:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/16 13:17:13 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 13:17:18 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 13:17:23 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 13:17:28 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 13:17:28 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/16 13:17:28 Starting bucket ops to source bucket
2021/12/16 13:17:49 Waiting for dst bucket item count to get to: 20000 curr count: 12679
2021/12/16 13:17:54 Waiting for dst bucket item count to get to: 20000 curr count: 17007
2021/12/16 13:17:54 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 13:17:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:17:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:18:07 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/16 13:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:18:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:18:19 Waiting for dst bucket item count to get to: 0 curr count: 18048
2021/12/16 13:18:24 Waiting for dst bucket item count to get to: 0 curr count: 8048
2021/12/16 13:18:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:18:24 Starting bucket ops to source bucket
2021/12/16 13:18:37 Waiting for dst bucket item count to get to: 10000 curr count: 2213
2021/12/16 13:18:42 Waiting for dst bucket item count to get to: 10000 curr count: 5800
2021/12/16 13:18:47 Waiting for dst bucket item count to get to: 10000 curr count: 8587
2021/12/16 13:18:47 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 13:18:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:18:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:18:52 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 13:18:57 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 13:19:02 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 13:19:07 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 13:19:07 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/16 13:19:07 No eventing-con process running
2021/12/16 13:19:07 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/16 13:19:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:19:18 Waiting for dst bucket item count to get to: 0 curr count: 7179
2021/12/16 13:19:23 Waiting for dst bucket item count to get to: 0 curr count: 2179
2021/12/16 13:19:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:19:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:19:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 13:19:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.75s)
=== RUN   TestTimerOverwrite
2021/12/16 13:19:45 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 13:19:45 App: TestTimerOverwrite got un-deployed
2021/12/16 13:19:45 No eventing-con process running
2021/12/16 13:19:45 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/16 13:19:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:19:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:19:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:19:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:19:58 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:19:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/16 13:20:04 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 13:20:09 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 13:20:14 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 13:20:19 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 13:20:19 App: TestTimerOverwrite got deployed
2021/12/16 13:20:29 Starting bucket ops to source bucket
2021/12/16 13:20:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:20:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:20:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:20:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:20:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:21:38 Waiting for dst bucket item count to get to: 5000 curr count: 2485
2021/12/16 13:21:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 13:21:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:21:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:21:53 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 13:21:58 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 13:22:03 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 13:22:08 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 13:22:08 App: TestTimerOverwrite got un-deployed
2021/12/16 13:22:08 No eventing-con process running
2021/12/16 13:22:08 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/16 13:22:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:22:19 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 13:22:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 13:22:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:22:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:22:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 13:22:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (181.48s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/16 13:22:41 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 13:22:41 Removing node(s): [] from the cluster
2021/12/16 13:22:47 Rebalance progress: 100
2021/12/16 13:22:47 Failing over: [127.0.0.1:19003]
2021/12/16 13:22:52 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 13:22:52 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 13:22:52 No eventing-con process running
2021/12/16 13:22:52 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/16 13:22:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:22:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:23:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:23:05 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:23:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 13:23:06 Starting bucket ops to source bucket
2021/12/16 13:23:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:23:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:23:16 Context: Post to set retry counter, response code: 200 dump: 
2021/12/16 13:23:16 set retry counter, response &{[] }
2021/12/16 13:24:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 13:24:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 13:24:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:24:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/16 13:24:26 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 13:24:26 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 13:24:26 No eventing-con process running
2021/12/16 13:24:26 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/16 13:24:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:24:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 13:24:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:24:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:24:49 Waiting for dst bucket item count to get to: 0 curr count: 2418
2021/12/16 13:24:54 Waiting for dst bucket item count to get to: 0 curr count: 752
2021/12/16 13:24:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:24:54 Removing node(s): [] from the cluster
2021/12/16 13:24:59 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.99s)
=== RUN   TestOnUpdateSrcMutation
2021/12/16 13:25:09 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 13:25:09 App: TestOnUpdateSrcMutation got un-deployed
2021/12/16 13:25:09 No eventing-con process running
2021/12/16 13:25:09 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/16 13:25:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:25:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:25:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:25:22 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:25:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/16 13:25:24 Starting bucket ops to source bucket
2021/12/16 13:25:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 13:25:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:25:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:25:51 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 13:25:56 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 13:26:01 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 13:26:06 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 13:26:06 App: TestOnUpdateSrcMutation got un-deployed
2021/12/16 13:26:06 No eventing-con process running
2021/12/16 13:26:06 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/16 13:26:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:26:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 13:26:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:26:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:26:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.59s)
=== RUN   TestOnDeleteSrcMutation
2021/12/16 13:26:34 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 13:26:34 App: TestOnDeleteSrcMutation got un-deployed
2021/12/16 13:26:34 No eventing-con process running
2021/12/16 13:26:34 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/16 13:26:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:26:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:26:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:26:47 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:26:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/16 13:26:48 Starting bucket ops to source bucket
2021/12/16 13:27:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 13:27:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:27:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:27:22 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 13:27:27 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 13:27:32 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 13:27:37 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 13:27:37 App: TestOnDeleteSrcMutation got un-deployed
2021/12/16 13:27:37 No eventing-con process running
2021/12/16 13:27:37 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/16 13:27:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:27:48 Waiting for dst bucket item count to get to: 0 curr count: 4809
2021/12/16 13:27:53 Waiting for dst bucket item count to get to: 0 curr count: 2309
2021/12/16 13:27:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.32s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/16 13:28:10 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 13:28:10 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/16 13:28:10 No eventing-con process running
2021/12/16 13:28:10 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/16 13:28:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:28:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:28:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:28:23 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:28:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 13:28:24 Starting bucket ops to source bucket
2021/12/16 13:29:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 13:29:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:29:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:29:12 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 13:29:17 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 13:29:22 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 13:29:27 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 13:29:27 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/16 13:29:27 No eventing-con process running
2021/12/16 13:29:27 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/16 13:29:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:29:38 Waiting for dst bucket item count to get to: 0 curr count: 6928
2021/12/16 13:29:43 Waiting for dst bucket item count to get to: 0 curr count: 1928
2021/12/16 13:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.60s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/16 13:30:00 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 13:30:00 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/16 13:30:00 No eventing-con process running
2021/12/16 13:30:00 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/16 13:30:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:30:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:30:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:30:13 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:30:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 13:30:14 Starting bucket ops to source bucket
2021/12/16 13:30:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 13:30:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:30:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:31:03 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 13:31:08 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 13:31:13 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 13:31:18 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 13:31:18 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/16 13:31:18 No eventing-con process running
2021/12/16 13:31:18 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/16 13:31:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:31:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 13:31:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 13:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:31:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:31:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.36s)
=== RUN   TestInterHandlerRecursion
2021/12/16 13:31:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:31:51 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 13:31:51 No eventing-con process running
2021/12/16 13:31:51 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/16 13:31:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:31:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:32:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:32:09 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:32:09 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 13:32:09 No eventing-con process running
2021/12/16 13:32:09 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 13:32:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:32:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:32:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:32:22 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:32:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/16 13:32:23 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:32:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:32:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:32:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:32:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:32:44 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/16 13:32:44 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:32:44 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/16 13:32:44 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/16 13:32:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:32:49 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:32:54 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:32:59 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:33:04 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:33:04 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 13:33:04 No eventing-con process running
2021/12/16 13:33:04 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/16 13:33:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:33:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:33:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:33:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:33:22 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:33:22 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 13:33:22 No eventing-con process running
2021/12/16 13:33:22 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 13:33:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:33:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:33:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.81s)
=== RUN   TestInterBucketRecursion
2021/12/16 13:33:40 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:33:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 13:33:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:33:41 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:33:41 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/16 13:33:46 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 13:33:51 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 13:33:56 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 13:34:01 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 13:34:01 App: TestInterBucketRecursion_function1 got deployed
2021/12/16 13:34:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:34:06 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:34:11 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:34:16 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:34:21 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:34:21 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/16 13:34:21 No eventing-con process running
2021/12/16 13:34:21 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/16 13:34:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:34:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:34:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:34:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:34:40 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/16 13:34:40 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/16 13:34:40 No eventing-con process running
2021/12/16 13:34:40 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/16 13:34:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:34:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:34:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.89s)
=== RUN   TestLargeHandler
2021/12/16 13:34:53 Changed force_compress value to false
2021/12/16 13:34:53 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:34:53 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/16 13:34:53 Changed force_compress value to true
2021/12/16 13:34:53 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:34:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/16 13:35:00 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 13:35:05 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 13:35:10 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 13:35:15 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 13:35:15 App: TestLargeHandler_function got deployed
2021/12/16 13:35:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:35:20 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 13:35:25 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 13:35:30 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 13:35:35 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 13:35:35 App: TestLargeHandler_function got un-deployed
2021/12/16 13:35:35 No eventing-con process running
2021/12/16 13:35:35 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/16 13:35:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:35:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:35:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:35:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.40s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/16 13:35:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:35:58 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 13:35:58 No eventing-con process running
2021/12/16 13:35:58 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/16 13:36:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:36:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:36:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:36:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:36:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 13:36:16 No eventing-con process running
2021/12/16 13:36:16 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 13:36:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:36:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:36:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:36:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:36:29 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:36:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/16 13:36:31 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:36:36 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:36:41 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:36:46 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:36:51 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 13:36:51 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/16 13:36:51 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:36:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/16 13:36:52 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:36:57 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 13:37:02 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 13:37:07 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 13:37:12 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 13:37:12 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/16 13:37:12 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/16 13:37:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:37:17 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:37:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:37:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:37:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 13:37:32 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 13:37:32 No eventing-con process running
2021/12/16 13:37:32 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/16 13:37:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:37:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:37:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:37:51 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:37:56 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:38:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:38:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 13:38:06 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 13:38:06 No eventing-con process running
2021/12/16 13:38:06 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/16 13:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:38:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.40s)
=== RUN   TestAllowInterBucketRecursion
2021/12/16 13:38:25 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:38:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 13:38:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:38:31 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 13:38:36 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 13:38:41 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 13:38:46 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 13:38:46 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/16 13:38:46 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:38:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 13:38:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:38:52 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 13:38:57 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 13:39:02 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 13:39:07 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 13:39:07 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/16 13:39:07 TestAllowInterBucketRecursion_function2 is deployed
2021/12/16 13:39:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:39:12 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:39:17 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:39:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:39:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 13:39:27 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/16 13:39:27 No eventing-con process running
2021/12/16 13:39:27 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/16 13:39:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:39:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:39:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:39:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:39:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:39:47 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 13:39:52 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 13:39:57 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 13:40:02 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 13:40:02 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/16 13:40:02 No eventing-con process running
2021/12/16 13:40:02 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/16 13:40:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:40:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.24s)
=== RUN   TestInterBucketRecursion2
2021/12/16 13:40:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:40:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:40:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:40:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:40:44 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:40:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 13:40:50 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 13:40:55 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 13:41:00 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 13:41:05 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 13:41:05 App: TestInterBucketRecursion2_function1 got deployed
2021/12/16 13:41:05 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:41:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 13:41:11 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 13:41:16 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 13:41:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 13:41:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 13:41:26 App: TestInterBucketRecursion2_function2 got deployed
2021/12/16 13:41:26 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:41:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 13:41:32 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 13:41:37 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 13:41:42 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 13:41:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 13:41:47 App: TestInterBucketRecursion2_function3 got deployed
2021/12/16 13:41:47 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:41:47 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/16 13:41:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:41:52 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 13:41:57 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 13:42:02 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 13:42:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 13:42:07 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/16 13:42:07 No eventing-con process running
2021/12/16 13:42:07 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/16 13:42:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:42:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:42:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:42:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:42:26 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 13:42:31 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 13:42:36 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 13:42:41 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 13:42:41 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/16 13:42:41 No eventing-con process running
2021/12/16 13:42:41 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/16 13:42:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:42:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:42:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:42:59 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 13:43:04 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 13:43:09 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 13:43:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 13:43:14 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/16 13:43:14 No eventing-con process running
2021/12/16 13:43:14 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/16 13:43:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:43:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:43:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:43:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:43:33 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/16 13:43:33 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/16 13:43:33 No eventing-con process running
2021/12/16 13:43:33 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/16 13:43:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:43:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:43:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (212.90s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/16 13:43:53 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 13:43:58 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/16 13:44:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:44:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:44:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:44:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:45:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:45:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:45:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:45:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:46:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:46:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:46:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:46:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:47:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:47:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:47:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:47:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:48:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:48:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:48:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:48:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:49:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:49:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:49:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 13:49:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/16 13:50:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:50:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 13:50:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 13:50:06 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/16 13:50:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:50:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 13:50:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 13:50:12 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/16 13:50:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:50:17 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/16 13:50:17 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/16 13:50:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:50:17 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/16 13:50:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:50:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:50:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:50:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:50:35 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:50:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 13:50:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 13:50:46 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 13:50:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 13:50:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 13:50:56 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: fcb202d91a8ed4ccd606f6d0b5ed4337
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: 1008c29742d57a2334ce9ce19d7f63b3
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 14baf2a7735e0ceb7e39eaeb73063ef9
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/16 13:50:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:50:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 13:51:01 Rebalance progress: 29.16666666666666
2021/12/16 13:51:06 Rebalance progress: 29.16666666666666
2021/12/16 13:51:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 14baf2a7735e0ceb7e39eaeb73063ef9
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: fcb202d91a8ed4ccd606f6d0b5ed4337
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: 1008c29742d57a2334ce9ce19d7f63b3
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]


2021/12/16 13:51:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:51:11 handler_test.go:1674: Deleting source bucket: default
2021/12/16 13:51:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 13:51:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 13:51:19 Rebalance progress: 29.16666666666666
2021/12/16 13:51:24 Rebalance progress: 29.16666666666666
2021/12/16 13:51:29 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 900
	vb list dump: [0, 2-5, 7-11, 13-28, 30-41, 43-48, 50-56, 58-63, 66-69, 71-88, 90-93, 95-100, 102-117, 121-123, 127-138, 140, 142-146, 148-153, 155-187, 189-196, 198-214, 216-229, 231-232, 234-254, 256-260, 263-266, 269, 271-272, 274-284, 286-290, 292-293, 295-296, 298-304, 306-328, 330-347, 349-356, 358-363, 365-370, 372-374, 376-382, 384-386, 388-400, 402-405, 407-410, 412-417, 419-426, 428, 430-433, 435-446, 448-460, 463, 465-467, 469-473, 475-483, 487-493, 496-499, 501, 503-506, 508, 510-582, 584-611, 613-627, 629-632, 634-635, 637-646, 649-654, 657-658, 660, 662-668, 670-681, 683-711, 713-719, 721-726, 728-741, 743-745, 747-756, 758-763, 766-767, 769, 771-782, 784-786, 788-792, 794-796, 798-805, 807-808, 811-815, 817-824, 826-832, 834-835, 837-842, 844-847, 849-858, 860-864, 866-875, 877-879, 881-898, 900-904, 906-912, 914-925, 927-947, 950-951, 953-964, 966-970, 972-989, 991, 993, 995-1006, 1008-1011, 1013-1023]
	stream status: stopped
	len: 124
	vb list dump: [1, 6, 12, 29, 42, 49, 57, 64-65, 70, 89, 94, 101, 118-120, 124-126, 139, 141, 147, 154, 188, 197, 215, 230, 233, 255, 261-262, 267-268, 270, 273, 285, 291, 294, 297, 305, 329, 348, 357, 364, 371, 375, 383, 387, 401, 406, 411, 418, 427, 429, 434, 447, 461-462, 464, 468, 474, 484-486, 494-495, 500, 502, 507, 509, 583, 612, 628, 633, 636, 647-648, 655-656, 659, 661, 669, 682, 712, 720, 727, 742, 746, 757, 764-765, 768, 770, 783, 787, 793, 797, 806, 809-810, 816, 825, 833, 836, 843, 848, 859, 865, 876, 880, 899, 905, 913, 926, 948-949, 952, 965, 971, 990, 992, 994, 1007, 1012]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: fcb202d91a8ed4ccd606f6d0b5ed4337
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-117, 121-123, 127-138, 140, 142-146, 148-153, 155-187, 189-196, 198-214, 216-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [228-229, 231-232, 234-254, 256-260, 263-266, 269, 271-272, 274-284, 286-290, 292-293, 295-296, 298-304, 306-328, 330-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0, 2-5, 7-11, 13-28, 30-41, 43-48, 50-56, 58-63, 66-69, 71-88, 90-93, 95-100, 102-113]
Producer node: 172.31.2.134:9300	Node UUID: 1008c29742d57a2334ce9ce19d7f63b3
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-912, 914-925, 927-947, 950-951, 953-964, 966-970, 972-989, 991, 993, 995-1006, 1008-1011, 1013-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-711, 713-719, 721-726, 728-741, 743-745, 747-756, 758-763, 766-767, 769, 771-782, 784-786, 788-792, 794-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 98
	v: [798-805, 807-808, 811-815, 817-824, 826-832, 834-835, 837-842, 844-847, 849-858, 860-864, 866-875, 877-879, 881-898, 900-904, 906-910]
Producer node: 127.0.0.1:9303	Node UUID: 14baf2a7735e0ceb7e39eaeb73063ef9
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [342-347, 349-356, 358-363, 365-370, 372-374, 376-382, 384-386, 388-400, 402-405, 407-410, 412-417, 419-426, 428, 430-433, 435-446, 448-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [456-460, 463, 465-467, 469-473, 475-483, 487-493, 496-499, 501, 503-506, 508, 510-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-582, 584-611, 613-627, 629-632, 634-635, 637-646, 649-654, 657-658, 660, 662-668, 670-681]


2021/12/16 13:51:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 13:51:34 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/16 13:51:34 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/16 13:51:39 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/16 13:51:39 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/16 13:51:39 eventingSetup.go:46: No eventing-con process running
2021/12/16 13:51:39 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/16 13:51:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:51:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:51:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:51:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:51:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 13:51:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 13:51:57 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (492.37s)
=== RUN   TestN1QLRecursion
2021/12/16 13:52:00 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 13:52:00 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 13:52:00 Removing node(s): [] from the cluster
2021/12/16 13:52:05 Rebalance progress: 100
2021/12/16 13:52:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:52:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:52:28 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:52:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 13:52:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:52:35 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 13:52:40 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 13:52:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 13:52:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 13:52:50 App: TestN1QLRecursion_function1 got deployed
2021/12/16 13:52:50 TestN1QLRecursion_function1 is deployed
2021/12/16 13:52:50 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:52:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 13:52:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:52:56 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 13:53:01 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 13:53:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 13:53:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 13:53:11 App: TestN1QLRecursion_function3 got deployed
2021/12/16 13:53:11 TestN1QLRecursion_function3 is deployed
2021/12/16 13:53:11 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:53:11 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/16 13:53:11 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/16 13:53:11 Success: TestN1QLRecursion_function2 is not deployed
2021/12/16 13:53:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:53:16 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 13:53:21 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 13:53:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 13:53:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 13:53:31 App: TestN1QLRecursion_function1 got un-deployed
2021/12/16 13:53:31 No eventing-con process running
2021/12/16 13:53:31 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/16 13:53:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:53:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:53:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:53:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:53:50 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 13:53:55 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 13:54:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 13:54:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 13:54:05 App: TestN1QLRecursion_function3 got un-deployed
2021/12/16 13:54:05 No eventing-con process running
2021/12/16 13:54:05 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/16 13:54:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:54:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:54:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:54:20 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 13:54:25 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.29s)
=== RUN   TestN1QLAllowRecursion
2021/12/16 13:54:25 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 13:54:26 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 13:54:26 Removing node(s): [] from the cluster
2021/12/16 13:54:31 Rebalance progress: 100
2021/12/16 13:54:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:54:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:54:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:54:54 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:54:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 13:54:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:55:00 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 13:55:05 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 13:55:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 13:55:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 13:55:15 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/16 13:55:15 TestN1QLAllowRecursion_function1 is deployed
2021/12/16 13:55:15 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:55:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 13:55:16 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:55:21 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 13:55:26 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 13:55:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 13:55:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 13:55:36 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/16 13:55:36 TestN1QLAllowRecursion_function3 is deployed
2021/12/16 13:55:36 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:55:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 13:55:37 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/16 13:55:42 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 13:55:47 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 13:55:53 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 13:55:58 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 13:55:58 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/16 13:55:58 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/16 13:55:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:56:03 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 13:56:08 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 13:56:13 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 13:56:18 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 13:56:18 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/16 13:56:18 No eventing-con process running
2021/12/16 13:56:18 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/16 13:56:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:56:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:56:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:56:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:56:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:56:37 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 13:56:42 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 13:56:47 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 13:56:52 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 13:56:52 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/16 13:56:52 No eventing-con process running
2021/12/16 13:56:52 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/16 13:56:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:56:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:57:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:57:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 13:57:11 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 13:57:16 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 13:57:21 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 13:57:26 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 13:57:26 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/16 13:57:26 No eventing-con process running
2021/12/16 13:57:26 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/16 13:57:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:57:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:57:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:57:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:57:41 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 13:57:46 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.77s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/16 13:57:46 Sleeping for some time
2021/12/16 13:57:56 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 13:57:56 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/16 13:57:56 No eventing-con process running
2021/12/16 13:57:56 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/16 13:57:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:58:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:58:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 13:58:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 13:58:10 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 13:58:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 13:58:11 Starting bucket ops to source bucket
2021/12/16 13:59:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:59:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 13:59:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 13:59:05 Starting bucket ops to source bucket
2021/12/16 13:59:06 Starting bucket ops to source bucket
2021/12/16 14:00:12 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/16 14:00:12 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/16 14:00:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:00:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:00:17 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 14:00:22 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 14:00:27 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 14:00:32 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 14:00:32 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/16 14:00:32 No eventing-con process running
2021/12/16 14:00:32 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/16 14:00:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:00:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:00:49 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/16 14:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:00:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:01:01 Waiting for dst bucket item count to get to: 0 curr count: 6927
2021/12/16 14:01:06 Waiting for dst bucket item count to get to: 0 curr count: 6927
2021/12/16 14:01:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.91s)
=== RUN   TestTimerOverWriteSameReference
2021/12/16 14:01:11 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 14:01:11 App: TestTimerOverWriteSameReference got un-deployed
2021/12/16 14:01:11 No eventing-con process running
2021/12/16 14:01:11 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/16 14:01:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:01:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:01:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:01:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:01:25 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:01:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/16 14:01:31 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 14:01:36 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 14:01:41 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 14:01:46 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 14:01:46 App: TestTimerOverWriteSameReference got deployed
2021/12/16 14:01:56 Starting bucket ops to source bucket
2021/12/16 14:02:57 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/16 14:03:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:03:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:03:12 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 14:03:17 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 14:03:22 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 14:03:27 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 14:03:27 App: TestTimerOverWriteSameReference got un-deployed
2021/12/16 14:03:27 No eventing-con process running
2021/12/16 14:03:27 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/16 14:03:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:03:39 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:03:44 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:03:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:03:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:03:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 14:03:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.82s)
=== RUN   TestCancelTimerBucketop
2021/12/16 14:04:01 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 14:04:01 App: TestCancelTimerBucketop got un-deployed
2021/12/16 14:04:01 No eventing-con process running
2021/12/16 14:04:01 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/16 14:04:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:04:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:04:15 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:04:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/16 14:04:21 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 14:04:26 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 14:04:31 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 14:04:36 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 14:04:36 App: TestCancelTimerBucketop got deployed
2021/12/16 14:04:36 Starting bucket ops to source bucket
2021/12/16 14:04:43 Starting bucket ops to source bucket
2021/12/16 14:05:45 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 14:05:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:05:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:05:50 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 14:05:55 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 14:06:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 14:06:06 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 14:06:06 App: TestCancelTimerBucketop got un-deployed
2021/12/16 14:06:06 No eventing-con process running
2021/12/16 14:06:06 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/16 14:06:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:06:17 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:06:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:06:29 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:06:34 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:06:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.31s)
=== RUN   TestJSExpiryDate
2021/12/16 14:06:39 Starting bucket ops to source bucket
2021/12/16 14:06:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 14:06:52 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/16 14:06:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/16 14:07:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:07:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:07:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:07:29 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 14:07:34 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 14:07:39 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 14:07:44 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 14:07:44 App: TestJSExpiryDate got un-deployed
2021/12/16 14:07:44 No eventing-con process running
2021/12/16 14:07:44 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/16 14:07:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:07:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:07:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:07:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.80s)
=== RUN   TestBinaryDoc
2021/12/16 14:08:03 Starting bucket ops to source bucket
2021/12/16 14:08:03 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:08:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/16 14:08:04 Starting bucket ops to source bucket
2021/12/16 14:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:08:04 Deleting source bucket docs now
2021/12/16 14:08:04 Starting bucket ops to source bucket
2021/12/16 14:08:41 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/16 14:08:46 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/16 14:08:46 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/16 14:08:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:08:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:08:52 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 14:08:57 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 14:09:02 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 14:09:07 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 14:09:07 App: TestBinaryDoc got un-deployed
2021/12/16 14:09:07 No eventing-con process running
2021/12/16 14:09:07 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/16 14:09:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:09:18 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/16 14:09:23 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/16 14:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:09:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:09:35 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/16 14:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.36s)
=== RUN   TestConstantBindings
2021/12/16 14:09:35 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:09:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/16 14:09:36 Starting bucket ops to source bucket
2021/12/16 14:09:42 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 14:09:47 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 14:09:52 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 14:09:57 Waiting for dst bucket item count to get to: 6000 curr count: 1601
2021/12/16 14:10:02 Waiting for dst bucket item count to get to: 6000 curr count: 1601
2021/12/16 14:10:02 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/16 14:10:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:10:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:10:07 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 14:10:12 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 14:10:17 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 14:10:22 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 14:10:22 App: TestConstantBindings got un-deployed
2021/12/16 14:10:22 No eventing-con process running
2021/12/16 14:10:22 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/16 14:10:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:10:34 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:10:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:10:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:10:51 Waiting for dst bucket item count to get to: 0 curr count: 5880
2021/12/16 14:10:56 Waiting for dst bucket item count to get to: 0 curr count: 5880
2021/12/16 14:10:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.71s)
=== RUN   TestCodeInsight
2021/12/16 14:11:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 14:11:01 App: TestCodeInsight got un-deployed
2021/12/16 14:11:01 No eventing-con process running
2021/12/16 14:11:01 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/16 14:11:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:11:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:11:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:11:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:11:15 Starting bucket ops to source bucket
2021/12/16 14:11:15 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:11:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/16 14:11:22 Waiting for app: TestCodeInsight to get deployed
2021/12/16 14:11:27 Waiting for app: TestCodeInsight to get deployed
2021/12/16 14:11:32 Waiting for app: TestCodeInsight to get deployed
2021/12/16 14:11:37 Waiting for app: TestCodeInsight to get deployed
2021/12/16 14:11:37 App: TestCodeInsight got deployed
2021/12/16 14:11:42 Waiting for dst bucket item count to get to: 1000 curr count: 326
2021/12/16 14:11:42 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 14:11:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:11:47 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 14:11:52 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 14:11:57 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 14:12:02 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 14:12:02 App: TestCodeInsight got un-deployed
2021/12/16 14:12:02 No eventing-con process running
2021/12/16 14:12:02 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/16 14:12:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:12:14 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:12:19 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:12:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:12:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:12:30 Waiting for dst bucket item count to get to: 0 curr count: 782
2021/12/16 14:12:35 Waiting for dst bucket item count to get to: 0 curr count: 782
2021/12/16 14:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.41s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.06s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13065.603s

Functional tests took 3 hours, 37 minutes and 49 seconds.

N1ql tests

2021/12/16 14:12:46 Started n1ql
2021/12/16 14:12:56 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 14:12:56 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 14:12:56 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 14:12:57 Removing node(s): [] from the cluster
2021/12/16 14:13:02 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/16 14:13:07 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/16 14:13:07 App: TestRecursiveMutationN1QL got un-deployed
2021/12/16 14:13:07 No eventing-con process running
2021/12/16 14:13:07 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/16 14:13:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:13:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:13:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:13:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:13:20 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:13:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.54s)
=== RUN   TestFlexReset1
2021/12/16 14:13:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 14:13:25 App: TestFlexReset1 got un-deployed
2021/12/16 14:13:25 No eventing-con process running
2021/12/16 14:13:25 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/16 14:13:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:13:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:13:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:13:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:13:39 Starting bucket ops to source bucket
2021/12/16 14:13:40 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:13:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/16 14:13:46 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 14:13:51 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 14:13:56 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 14:14:01 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 14:14:01 App: TestFlexReset1 got deployed
2021/12/16 14:14:06 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/16 14:14:06 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 14:14:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:14:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:14:11 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 14:14:16 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 14:14:21 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 14:14:26 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 14:14:26 App: TestFlexReset1 got un-deployed
2021/12/16 14:14:26 No eventing-con process running
2021/12/16 14:14:26 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/16 14:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:14:38 Waiting for dst bucket item count to get to: 0 curr count: 571
2021/12/16 14:14:43 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/16 14:14:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:14:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:14:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:14:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.31s)
=== RUN   TestFlexReset2
2021/12/16 14:15:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 14:15:00 App: TestFlexReset2 got un-deployed
2021/12/16 14:15:00 No eventing-con process running
2021/12/16 14:15:00 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/16 14:15:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:15:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:15:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:15:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:15:13 Starting bucket ops to source bucket
2021/12/16 14:15:14 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:15:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/16 14:15:20 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 14:15:25 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 14:15:30 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 14:15:35 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 14:15:35 App: TestFlexReset2 got deployed
2021/12/16 14:15:40 Waiting for dst bucket item count to get to: 1000 curr count: 380
2021/12/16 14:15:45 Waiting for dst bucket item count to get to: 1000 curr count: 859
2021/12/16 14:15:45 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 14:15:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:15:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:15:50 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 14:15:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 14:16:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 14:16:05 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 14:16:05 App: TestFlexReset2 got un-deployed
2021/12/16 14:16:05 No eventing-con process running
2021/12/16 14:16:05 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/16 14:16:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:16:17 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 14:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:16:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:16:29 Waiting for dst bucket item count to get to: 0 curr count: 964
2021/12/16 14:16:34 Waiting for dst bucket item count to get to: 0 curr count: 464
2021/12/16 14:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.27s)
=== RUN   TestN1QLLabelledBreak
2021/12/16 14:16:39 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 14:16:39 App: TestN1QLLabelledBreak got un-deployed
2021/12/16 14:16:39 No eventing-con process running
2021/12/16 14:16:39 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/16 14:16:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:16:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:16:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:16:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:16:52 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:16:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/16 14:16:59 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 14:17:04 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 14:17:09 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 14:17:14 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 14:17:14 App: TestN1QLLabelledBreak got deployed
2021/12/16 14:17:14 Starting bucket ops to source bucket
2021/12/16 14:17:23 Waiting for dst bucket item count to get to: 10000 curr count: 236
2021/12/16 14:17:28 Waiting for dst bucket item count to get to: 10000 curr count: 236
2021/12/16 14:17:33 Waiting for dst bucket item count to get to: 10000 curr count: 729
2021/12/16 14:17:38 Waiting for dst bucket item count to get to: 10000 curr count: 2588
2021/12/16 14:17:43 Waiting for dst bucket item count to get to: 10000 curr count: 4628
2021/12/16 14:17:48 Waiting for dst bucket item count to get to: 10000 curr count: 6678
2021/12/16 14:17:53 Waiting for dst bucket item count to get to: 10000 curr count: 8693
2021/12/16 14:17:58 Waiting for dst bucket item count to get to: 10000 curr count: 9548
2021/12/16 14:17:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 14:17:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:17:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:18:03 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 14:18:08 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 14:18:13 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 14:18:18 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 14:18:18 App: TestN1QLLabelledBreak got un-deployed
2021/12/16 14:18:18 No eventing-con process running
2021/12/16 14:18:19 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/16 14:18:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:18:30 Waiting for dst bucket item count to get to: 0 curr count: 3606
2021/12/16 14:18:35 Waiting for dst bucket item count to get to: 0 curr count: 3606
2021/12/16 14:18:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:18:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:18:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 14:18:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.29s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/16 14:18:52 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 14:18:52 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/16 14:18:52 No eventing-con process running
2021/12/16 14:18:52 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/16 14:18:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:18:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:19:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:19:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:19:06 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:19:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/16 14:19:12 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 14:19:17 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 14:19:22 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 14:19:27 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 14:19:27 App: TestN1QLUnlabelledBreak got deployed
2021/12/16 14:19:27 Starting bucket ops to source bucket
2021/12/16 14:19:36 Waiting for dst bucket item count to get to: 10000 curr count: 258
2021/12/16 14:19:41 Waiting for dst bucket item count to get to: 10000 curr count: 258
2021/12/16 14:19:46 Waiting for dst bucket item count to get to: 10000 curr count: 258
2021/12/16 14:19:51 Waiting for dst bucket item count to get to: 10000 curr count: 4878
2021/12/16 14:19:56 Waiting for dst bucket item count to get to: 10000 curr count: 4878
2021/12/16 14:20:01 Waiting for dst bucket item count to get to: 10000 curr count: 9969
2021/12/16 14:20:06 Waiting for dst bucket item count to get to: 10000 curr count: 9969
2021/12/16 14:20:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 14:20:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:20:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:20:12 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 14:20:17 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 14:20:22 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 14:20:27 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 14:20:27 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/16 14:20:27 No eventing-con process running
2021/12/16 14:20:27 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/16 14:20:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:20:39 Waiting for dst bucket item count to get to: 0 curr count: 4198
2021/12/16 14:20:44 Waiting for dst bucket item count to get to: 0 curr count: 1698
2021/12/16 14:20:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:20:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:20:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 14:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.28s)
=== RUN   TestN1QLThrowStatement
2021/12/16 14:21:00 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 14:21:00 App: TestN1QLThrowStatement got un-deployed
2021/12/16 14:21:00 No eventing-con process running
2021/12/16 14:21:00 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/16 14:21:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:21:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:21:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:21:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:21:14 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:21:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/16 14:21:20 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 14:21:25 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 14:21:30 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 14:21:35 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 14:21:35 App: TestN1QLThrowStatement got deployed
2021/12/16 14:21:35 Starting bucket ops to source bucket
2021/12/16 14:21:45 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 14:21:50 Waiting for dst bucket item count to get to: 10000 curr count: 412
2021/12/16 14:21:55 Waiting for dst bucket item count to get to: 10000 curr count: 412
2021/12/16 14:22:00 Waiting for dst bucket item count to get to: 10000 curr count: 4098
2021/12/16 14:22:05 Waiting for dst bucket item count to get to: 10000 curr count: 4098
2021/12/16 14:22:10 Waiting for dst bucket item count to get to: 10000 curr count: 8139
2021/12/16 14:22:15 Waiting for dst bucket item count to get to: 10000 curr count: 8139
2021/12/16 14:22:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 14:22:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:22:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:22:20 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 14:22:25 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 14:22:30 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 14:22:35 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 14:22:35 App: TestN1QLThrowStatement got un-deployed
2021/12/16 14:22:35 No eventing-con process running
2021/12/16 14:22:35 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/16 14:22:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:22:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 14:22:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:22:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:22:59 Waiting for dst bucket item count to get to: 0 curr count: 8842
2021/12/16 14:23:04 Waiting for dst bucket item count to get to: 0 curr count: 3842
2021/12/16 14:23:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.15s)
=== RUN   TestN1QLNestedForLoop
2021/12/16 14:23:09 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 14:23:09 App: TestN1QLNestedForLoop got un-deployed
2021/12/16 14:23:09 No eventing-con process running
2021/12/16 14:23:09 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/16 14:23:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:23:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:23:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:23:22 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:23:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/16 14:23:28 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 14:23:33 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 14:23:38 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 14:23:43 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 14:23:43 App: TestN1QLNestedForLoop got deployed
2021/12/16 14:23:43 Starting bucket ops to source bucket
2021/12/16 14:23:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 14:23:58 Waiting for dst bucket item count to get to: 5000 curr count: 144
2021/12/16 14:24:03 Waiting for dst bucket item count to get to: 5000 curr count: 482
2021/12/16 14:24:08 Waiting for dst bucket item count to get to: 5000 curr count: 1629
2021/12/16 14:24:13 Waiting for dst bucket item count to get to: 5000 curr count: 2840
2021/12/16 14:24:18 Waiting for dst bucket item count to get to: 5000 curr count: 4039
2021/12/16 14:24:23 Waiting for dst bucket item count to get to: 5000 curr count: 4990
2021/12/16 14:24:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 14:24:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:24:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:24:28 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 14:24:33 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 14:24:38 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 14:24:43 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 14:24:43 App: TestN1QLNestedForLoop got un-deployed
2021/12/16 14:24:43 No eventing-con process running
2021/12/16 14:24:43 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/16 14:24:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:24:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 14:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:24:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:25:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 14:25:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.97s)
=== RUN   TestN1QLAutoClose
2021/12/16 14:25:12 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 14:25:12 App: TestN1QLAutoClose got un-deployed
2021/12/16 14:25:12 No eventing-con process running
2021/12/16 14:25:12 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/16 14:25:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:25:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:25:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:25:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:25:25 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:25:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/16 14:25:31 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 14:25:36 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 14:25:41 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 14:25:46 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 14:25:46 App: TestN1QLAutoClose got deployed
2021/12/16 14:25:46 Starting bucket ops to source bucket
2021/12/16 14:25:52 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/16 14:25:57 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/16 14:26:02 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/16 14:26:07 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/16 14:26:12 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/16 14:26:17 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/16 14:26:22 Waiting for dst bucket item count to get to: 500 curr count: 46
2021/12/16 14:26:27 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/16 14:26:32 Waiting for dst bucket item count to get to: 500 curr count: 67
2021/12/16 14:26:37 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/16 14:26:42 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/16 14:26:47 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/16 14:26:52 Waiting for dst bucket item count to get to: 500 curr count: 111
2021/12/16 14:26:57 Waiting for dst bucket item count to get to: 500 curr count: 123
2021/12/16 14:27:02 Waiting for dst bucket item count to get to: 500 curr count: 132
2021/12/16 14:27:07 Waiting for dst bucket item count to get to: 500 curr count: 132
2021/12/16 14:27:12 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/16 14:27:17 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/16 14:27:22 Waiting for dst bucket item count to get to: 500 curr count: 175
2021/12/16 14:27:27 Waiting for dst bucket item count to get to: 500 curr count: 187
2021/12/16 14:27:32 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/16 14:27:37 Waiting for dst bucket item count to get to: 500 curr count: 208
2021/12/16 14:27:42 Waiting for dst bucket item count to get to: 500 curr count: 219
2021/12/16 14:27:47 Waiting for dst bucket item count to get to: 500 curr count: 229
2021/12/16 14:27:52 Waiting for dst bucket item count to get to: 500 curr count: 240
2021/12/16 14:27:57 Waiting for dst bucket item count to get to: 500 curr count: 250
2021/12/16 14:28:02 Waiting for dst bucket item count to get to: 500 curr count: 261
2021/12/16 14:28:07 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/16 14:28:12 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/16 14:28:17 Waiting for dst bucket item count to get to: 500 curr count: 296
2021/12/16 14:28:22 Waiting for dst bucket item count to get to: 500 curr count: 305
2021/12/16 14:28:27 Waiting for dst bucket item count to get to: 500 curr count: 317
2021/12/16 14:28:32 Waiting for dst bucket item count to get to: 500 curr count: 325
2021/12/16 14:28:37 Waiting for dst bucket item count to get to: 500 curr count: 337
2021/12/16 14:28:42 Waiting for dst bucket item count to get to: 500 curr count: 347
2021/12/16 14:28:47 Waiting for dst bucket item count to get to: 500 curr count: 359
2021/12/16 14:28:52 Waiting for dst bucket item count to get to: 500 curr count: 368
2021/12/16 14:28:57 Waiting for dst bucket item count to get to: 500 curr count: 377
2021/12/16 14:29:02 Waiting for dst bucket item count to get to: 500 curr count: 387
2021/12/16 14:29:07 Waiting for dst bucket item count to get to: 500 curr count: 410
2021/12/16 14:29:12 Waiting for dst bucket item count to get to: 500 curr count: 410
2021/12/16 14:29:17 Waiting for dst bucket item count to get to: 500 curr count: 430
2021/12/16 14:29:22 Waiting for dst bucket item count to get to: 500 curr count: 430
2021/12/16 14:29:27 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/16 14:29:32 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/16 14:29:37 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/16 14:29:42 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/16 14:29:47 Waiting for dst bucket item count to get to: 500 curr count: 495
2021/12/16 14:29:52 Waiting for dst bucket item count to get to: 500 curr count: 495
2021/12/16 14:29:52 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/16 14:29:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:29:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:29:58 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 14:30:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 14:30:08 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 14:30:13 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 14:30:13 App: TestN1QLAutoClose got un-deployed
2021/12/16 14:30:13 No eventing-con process running
2021/12/16 14:30:13 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/16 14:30:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:30:25 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:30:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:30:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:30:36 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 14:30:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.90s)
=== RUN   TestN1QLPosParams
2021/12/16 14:30:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 14:30:41 App: TestN1QLPosParams got un-deployed
2021/12/16 14:30:41 No eventing-con process running
2021/12/16 14:30:41 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/16 14:30:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:30:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:30:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:30:55 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:30:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/16 14:31:01 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 14:31:06 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 14:31:11 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 14:31:16 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 14:31:16 App: TestN1QLPosParams got deployed
2021/12/16 14:31:16 Starting bucket ops to source bucket
2021/12/16 14:31:26 Waiting for dst bucket item count to get to: 5000 curr count: 641
2021/12/16 14:31:31 Waiting for dst bucket item count to get to: 5000 curr count: 641
2021/12/16 14:31:31 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 14:31:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:31:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:31:36 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 14:31:41 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 14:31:46 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 14:31:51 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 14:31:51 App: TestN1QLPosParams got un-deployed
2021/12/16 14:31:51 No eventing-con process running
2021/12/16 14:31:51 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/16 14:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:32:03 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 14:32:08 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 14:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:32:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:32:20 Waiting for dst bucket item count to get to: 0 curr count: 2564
2021/12/16 14:32:25 Waiting for dst bucket item count to get to: 0 curr count: 2564
2021/12/16 14:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.38s)
=== RUN   TestN1QLExhaustConnPool
2021/12/16 14:32:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 14:32:30 App: TestN1QLExhaustConnPool got un-deployed
2021/12/16 14:32:30 No eventing-con process running
2021/12/16 14:32:30 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/16 14:32:32 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:32:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:32:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:32:43 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:32:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/16 14:32:50 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 14:32:55 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 14:33:00 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 14:33:05 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 14:33:05 App: TestN1QLExhaustConnPool got deployed
2021/12/16 14:33:05 Starting bucket ops to source bucket
2021/12/16 14:33:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 14:33:15 Waiting for dst bucket item count to get to: 100 curr count: 34
2021/12/16 14:33:20 Waiting for dst bucket item count to get to: 100 curr count: 84
2021/12/16 14:33:20 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 14:33:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:33:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:33:25 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 14:33:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 14:33:35 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 14:33:40 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 14:33:40 App: TestN1QLExhaustConnPool got un-deployed
2021/12/16 14:33:40 No eventing-con process running
2021/12/16 14:33:40 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/16 14:33:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:33:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 14:33:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 14:33:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:33:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:34:09 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/16 14:34:14 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/16 14:34:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.90s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/16 14:34:19 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 14:34:19 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/16 14:34:19 No eventing-con process running
2021/12/16 14:34:19 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/16 14:34:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:34:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:34:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:34:32 Starting bucket ops to source bucket
2021/12/16 14:34:50 Starting bucket ops to source bucket
2021/12/16 14:35:08 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 14232
2021/12/16 14:35:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/16 14:35:14 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 14:35:19 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 14:35:24 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 14:35:29 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 14:35:29 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/16 14:35:29 Starting bucket ops to source bucket
2021/12/16 14:35:34 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:35:39 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:35:44 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:35:49 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:35:54 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:35:59 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:36:04 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:36:09 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:36:14 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:36:19 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:36:24 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 14:36:24 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/16 14:36:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:36:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:36:29 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 14:36:34 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 14:36:39 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 14:36:44 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 14:36:44 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/16 14:36:44 No eventing-con process running
2021/12/16 14:36:44 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/16 14:36:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:36:56 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/16 14:36:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:36:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:37:08 Waiting for dst bucket item count to get to: 0 curr count: 20247
2021/12/16 14:37:13 Waiting for dst bucket item count to get to: 0 curr count: 5247
2021/12/16 14:37:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.24s)
=== RUN   TestN1QLQueryConsistency
2021/12/16 14:37:18 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 14:37:18 App: n1qlquery_consistency got un-deployed
2021/12/16 14:37:18 No eventing-con process running
2021/12/16 14:37:18 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/16 14:37:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:37:32 Starting bucket ops to source bucket
2021/12/16 14:37:35 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:37:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/16 14:37:41 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 14:37:46 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 14:37:51 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 14:37:56 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 14:37:56 App: n1qlquery_consistency got deployed
2021/12/16 14:38:01 Waiting for dst bucket item count to get to: 5000 curr count: 302
2021/12/16 14:38:06 Waiting for dst bucket item count to get to: 5000 curr count: 302
2021/12/16 14:38:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 14:38:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 14:38:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:38:11 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 14:38:16 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 14:38:21 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 14:38:26 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 14:38:26 App: n1qlquery_consistency got un-deployed
2021/12/16 14:38:26 No eventing-con process running
2021/12/16 14:38:26 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/16 14:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:38:38 Waiting for dst bucket item count to get to: 0 curr count: 2766
2021/12/16 14:38:43 Waiting for dst bucket item count to get to: 0 curr count: 266
2021/12/16 14:38:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:38:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 14:38:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 14:38:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.61s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1564.313s

N1ql tests took 0 hours, 26 minutes and 9 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/16 14:40:30 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/16 14:40:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/16 14:40:53 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 14:40:53 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/16 14:40:53 No eventing-con process running
2021/12/16 14:40:53 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/16 14:40:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:41:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:41:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:41:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:41:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:41:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:41:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:41:11 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:41:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 14:41:17 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 14:41:22 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 14:41:27 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 14:41:32 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 14:41:32 App: TestDuplicateWithEventingRebIn got deployed
2021/12/16 14:41:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 14:41:32 Starting bucket ops to source bucket
2021/12/16 14:41:47 Removing node(s): [] from the cluster
2021/12/16 14:41:52 Rebalance progress: 0
2021/12/16 14:41:57 Rebalance progress: 0
2021/12/16 14:42:02 Rebalance progress: 25
2021/12/16 14:42:07 Rebalance progress: 25
2021/12/16 14:42:12 Rebalance progress: 25
2021/12/16 14:42:17 Rebalance progress: 97.11538461538463
2021/12/16 14:42:22 Rebalance progress: 97.11538461538463
2021/12/16 14:42:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:42:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:42:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 14:42:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:42:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 14:42:32 Rebalance progress: 25
2021/12/16 14:42:37 Rebalance progress: 25
2021/12/16 14:42:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:42:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:42:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:42:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 14:42:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:42:42 Waiting for src bucket item count to get to: 200000 curr count: 89235
2021/12/16 14:42:47 Waiting for src bucket item count to get to: 200000 curr count: 105994
2021/12/16 14:42:52 Waiting for src bucket item count to get to: 200000 curr count: 105994
2021/12/16 14:42:57 Waiting for src bucket item count to get to: 200000 curr count: 122414
2021/12/16 14:43:02 Waiting for src bucket item count to get to: 200000 curr count: 122414
2021/12/16 14:43:07 Waiting for src bucket item count to get to: 200000 curr count: 139017
2021/12/16 14:43:12 Waiting for src bucket item count to get to: 200000 curr count: 139017
2021/12/16 14:43:17 Waiting for src bucket item count to get to: 200000 curr count: 155466
2021/12/16 14:43:22 Waiting for src bucket item count to get to: 200000 curr count: 155466
2021/12/16 14:43:27 Waiting for src bucket item count to get to: 200000 curr count: 171587
2021/12/16 14:43:32 Waiting for src bucket item count to get to: 200000 curr count: 171587
2021/12/16 14:43:37 Waiting for src bucket item count to get to: 200000 curr count: 187840
2021/12/16 14:43:42 Waiting for src bucket item count to get to: 200000 curr count: 187840
2021/12/16 14:43:52 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:43:57 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:02 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:07 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:12 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:17 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:22 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:27 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:32 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:37 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 14:44:37 Post SET ops. Expected item count: 200000 got 200001
2021/12/16 14:44:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:44:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 14:44:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 14:44:52 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 14:44:57 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 14:44:57 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/16 14:44:57 No eventing-con process running
2021/12/16 14:44:57 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/16 14:44:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:45:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:45:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 14:45:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:45:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:45:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:45:20 Waiting for dst bucket item count to get to: 0 curr count: 97736
2021/12/16 14:45:25 Waiting for dst bucket item count to get to: 0 curr count: 97736
2021/12/16 14:45:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (287.61s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/16 14:45:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 14:45:31 Removing node(s): [] from the cluster
2021/12/16 14:45:36 Rebalance progress: 100
2021/12/16 14:45:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 14:45:41 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/16 14:45:41 No eventing-con process running
2021/12/16 14:45:41 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/16 14:45:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:45:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:45:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:45:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:45:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:45:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:45:59 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:46:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 14:46:05 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 14:46:10 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 14:46:15 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 14:46:20 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 14:46:20 App: TestDuplicateWithEventingRebOut got deployed
2021/12/16 14:46:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 14:46:20 Starting bucket ops to source bucket
2021/12/16 14:46:26 Rebalance progress: 25
2021/12/16 14:46:31 Rebalance progress: 25
2021/12/16 14:46:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:46:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 14:46:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:46:36 Waiting for src bucket item count to get to: 200000 curr count: 14097
2021/12/16 14:46:41 Waiting for src bucket item count to get to: 200000 curr count: 30546
2021/12/16 14:46:46 Waiting for src bucket item count to get to: 200000 curr count: 30546
2021/12/16 14:46:51 Waiting for src bucket item count to get to: 200000 curr count: 47145
2021/12/16 14:46:56 Waiting for src bucket item count to get to: 200000 curr count: 47145
2021/12/16 14:47:01 Waiting for src bucket item count to get to: 200000 curr count: 63846
2021/12/16 14:47:06 Waiting for src bucket item count to get to: 200000 curr count: 63846
2021/12/16 14:47:11 Waiting for src bucket item count to get to: 200000 curr count: 63846
2021/12/16 14:47:16 Waiting for src bucket item count to get to: 200000 curr count: 79999
2021/12/16 14:47:21 Waiting for src bucket item count to get to: 200000 curr count: 79999
2021/12/16 14:47:26 Waiting for src bucket item count to get to: 200000 curr count: 96307
2021/12/16 14:47:31 Waiting for src bucket item count to get to: 200000 curr count: 96307
2021/12/16 14:47:36 Waiting for src bucket item count to get to: 200000 curr count: 112887
2021/12/16 14:47:41 Waiting for src bucket item count to get to: 200000 curr count: 112887
2021/12/16 14:47:46 Waiting for src bucket item count to get to: 200000 curr count: 129559
2021/12/16 14:47:51 Waiting for src bucket item count to get to: 200000 curr count: 129559
2021/12/16 14:47:56 Waiting for src bucket item count to get to: 200000 curr count: 146167
2021/12/16 14:48:01 Waiting for src bucket item count to get to: 200000 curr count: 146167
2021/12/16 14:48:06 Waiting for src bucket item count to get to: 200000 curr count: 162799
2021/12/16 14:48:11 Waiting for src bucket item count to get to: 200000 curr count: 162799
2021/12/16 14:48:16 Waiting for src bucket item count to get to: 200000 curr count: 178733
2021/12/16 14:48:21 Waiting for src bucket item count to get to: 200000 curr count: 178733
2021/12/16 14:48:26 Waiting for src bucket item count to get to: 200000 curr count: 194892
2021/12/16 14:48:31 Waiting for src bucket item count to get to: 200000 curr count: 194892
2021/12/16 14:48:36 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 14:48:36 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 14:48:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:48:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 14:48:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 14:48:51 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 14:48:56 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 14:48:56 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/16 14:48:56 No eventing-con process running
2021/12/16 14:48:56 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/16 14:48:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:49:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:49:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 14:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:49:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:49:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:49:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 14:49:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (233.89s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/16 14:49:29 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 14:49:29 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/16 14:49:29 No eventing-con process running
2021/12/16 14:49:29 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/16 14:49:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:49:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:49:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:49:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:49:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:49:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:49:48 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:49:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 14:49:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 14:49:59 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 14:50:04 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 14:50:09 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 14:50:09 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/16 14:50:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 14:50:09 Starting bucket ops to source bucket
2021/12/16 14:50:10 Removing node(s): [] from the cluster
2021/12/16 14:50:15 Rebalance progress: 25
2021/12/16 14:50:20 Rebalance progress: 25
2021/12/16 14:50:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:50:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 14:50:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:50:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 14:50:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 14:50:36 Rebalance progress: 16.66666666666667
2021/12/16 14:50:41 Rebalance progress: 16.66666666666667
2021/12/16 14:50:46 Rebalance progress: 16.66666666666667
2021/12/16 14:50:51 Rebalance progress: 32.26381461675579
2021/12/16 14:50:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:50:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:50:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:50:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 14:51:02 Rebalance progress: 16.66666666666667
2021/12/16 14:51:06 Rebalance progress: 16.66666666666667
2021/12/16 14:51:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:51:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:51:12 Waiting for src bucket item count to get to: 200000 curr count: 74615
2021/12/16 14:51:17 Waiting for src bucket item count to get to: 200000 curr count: 91080
2021/12/16 14:51:22 Waiting for src bucket item count to get to: 200000 curr count: 91080
2021/12/16 14:51:27 Waiting for src bucket item count to get to: 200000 curr count: 107425
2021/12/16 14:51:32 Waiting for src bucket item count to get to: 200000 curr count: 123882
2021/12/16 14:51:37 Waiting for src bucket item count to get to: 200000 curr count: 123882
2021/12/16 14:51:42 Waiting for src bucket item count to get to: 200000 curr count: 140073
2021/12/16 14:51:47 Waiting for src bucket item count to get to: 200000 curr count: 140073
2021/12/16 14:51:52 Waiting for src bucket item count to get to: 200000 curr count: 156346
2021/12/16 14:51:57 Waiting for src bucket item count to get to: 200000 curr count: 156346
2021/12/16 14:52:02 Waiting for src bucket item count to get to: 200000 curr count: 172112
2021/12/16 14:52:07 Waiting for src bucket item count to get to: 200000 curr count: 172112
2021/12/16 14:52:12 Waiting for src bucket item count to get to: 200000 curr count: 188359
2021/12/16 14:52:17 Waiting for src bucket item count to get to: 200000 curr count: 188359
2021/12/16 14:52:22 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 14:52:22 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 14:52:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:52:27 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 14:52:32 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 14:52:37 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 14:52:42 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 14:52:42 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/16 14:52:42 No eventing-con process running
2021/12/16 14:52:42 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/16 14:52:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:52:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:52:53 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 14:52:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 14:52:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:53:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:53:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:53:10 Waiting for dst bucket item count to get to: 0 curr count: 68022
2021/12/16 14:53:15 Waiting for dst bucket item count to get to: 0 curr count: 68022
2021/12/16 14:53:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (235.98s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/16 14:53:25 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 14:53:25 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/16 14:53:25 No eventing-con process running
2021/12/16 14:53:25 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/16 14:53:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:53:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:53:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:53:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:53:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:53:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:53:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:53:44 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:53:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 14:53:50 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 14:53:55 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 14:54:00 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 14:54:05 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 14:54:05 App: TestDuplicateWithKVRebIn got deployed
2021/12/16 14:54:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 14:54:05 Starting bucket ops to source bucket
2021/12/16 14:54:06 Removing node(s): [] from the cluster
2021/12/16 14:54:11 Rebalance progress: 16.66666666666667
2021/12/16 14:54:16 Rebalance progress: 16.66666666666667
2021/12/16 14:54:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:54:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:54:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:54:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:54:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 14:54:26 Rebalance progress: 16.66666666666667
2021/12/16 14:54:31 Rebalance progress: 16.66666666666667
2021/12/16 14:54:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:54:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:54:36 Waiting for src bucket item count to get to: 200000 curr count: 37278
2021/12/16 14:54:41 Waiting for src bucket item count to get to: 200000 curr count: 53245
2021/12/16 14:54:46 Waiting for src bucket item count to get to: 200000 curr count: 53245
2021/12/16 14:54:51 Waiting for src bucket item count to get to: 200000 curr count: 69712
2021/12/16 14:54:56 Waiting for src bucket item count to get to: 200000 curr count: 69712
2021/12/16 14:55:01 Waiting for src bucket item count to get to: 200000 curr count: 85007
2021/12/16 14:55:06 Waiting for src bucket item count to get to: 200000 curr count: 85007
2021/12/16 14:55:11 Waiting for src bucket item count to get to: 200000 curr count: 101368
2021/12/16 14:55:16 Waiting for src bucket item count to get to: 200000 curr count: 101368
2021/12/16 14:55:21 Waiting for src bucket item count to get to: 200000 curr count: 117947
2021/12/16 14:55:26 Waiting for src bucket item count to get to: 200000 curr count: 117947
2021/12/16 14:55:31 Waiting for src bucket item count to get to: 200000 curr count: 134339
2021/12/16 14:55:36 Waiting for src bucket item count to get to: 200000 curr count: 134339
2021/12/16 14:55:41 Waiting for src bucket item count to get to: 200000 curr count: 134339
2021/12/16 14:55:46 Waiting for src bucket item count to get to: 200000 curr count: 150813
2021/12/16 14:55:51 Waiting for src bucket item count to get to: 200000 curr count: 150813
2021/12/16 14:55:56 Waiting for src bucket item count to get to: 200000 curr count: 167349
2021/12/16 14:56:01 Waiting for src bucket item count to get to: 200000 curr count: 167349
2021/12/16 14:56:06 Waiting for src bucket item count to get to: 200000 curr count: 182727
2021/12/16 14:56:11 Waiting for src bucket item count to get to: 200000 curr count: 182727
2021/12/16 14:56:16 Waiting for src bucket item count to get to: 200000 curr count: 199063
2021/12/16 14:56:21 Waiting for src bucket item count to get to: 200000 curr count: 199063
2021/12/16 14:56:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 14:56:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 14:56:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 14:56:31 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 14:56:36 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 14:56:41 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 14:56:46 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 14:56:46 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/16 14:56:46 No eventing-con process running
2021/12/16 14:56:46 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/16 14:56:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:56:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 14:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:57:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:57:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.50s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/16 14:57:10 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 14:57:10 Removing node(s): [] from the cluster
2021/12/16 14:57:15 Rebalance progress: 100
2021/12/16 14:57:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 14:57:20 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/16 14:57:20 No eventing-con process running
2021/12/16 14:57:20 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/16 14:57:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 14:57:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:57:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:57:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 14:57:39 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 14:57:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 14:57:45 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 14:57:50 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 14:57:55 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 14:57:55 App: TestDuplicateWithKVRebOut got deployed
2021/12/16 14:57:55 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 14:57:55 Starting bucket ops to source bucket
2021/12/16 14:58:00 Rebalance progress: 16.66666666666667
2021/12/16 14:58:05 Rebalance progress: 16.66666666666667
2021/12/16 14:58:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 14:58:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 14:58:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 14:58:10 Waiting for src bucket item count to get to: 200000 curr count: 6471
2021/12/16 14:58:15 Waiting for src bucket item count to get to: 200000 curr count: 22653
2021/12/16 14:58:20 Waiting for src bucket item count to get to: 200000 curr count: 38624
2021/12/16 14:58:25 Waiting for src bucket item count to get to: 200000 curr count: 38624
2021/12/16 14:58:30 Waiting for src bucket item count to get to: 200000 curr count: 54864
2021/12/16 14:58:35 Waiting for src bucket item count to get to: 200000 curr count: 54864
2021/12/16 14:58:40 Waiting for src bucket item count to get to: 200000 curr count: 71119
2021/12/16 14:58:45 Waiting for src bucket item count to get to: 200000 curr count: 71119
2021/12/16 14:58:50 Waiting for src bucket item count to get to: 200000 curr count: 71119
2021/12/16 14:58:55 Waiting for src bucket item count to get to: 200000 curr count: 86558
2021/12/16 14:59:00 Waiting for src bucket item count to get to: 200000 curr count: 86558
2021/12/16 14:59:05 Waiting for src bucket item count to get to: 200000 curr count: 102686
2021/12/16 14:59:10 Waiting for src bucket item count to get to: 200000 curr count: 102686
2021/12/16 14:59:15 Waiting for src bucket item count to get to: 200000 curr count: 119155
2021/12/16 14:59:20 Waiting for src bucket item count to get to: 200000 curr count: 119155
2021/12/16 14:59:25 Waiting for src bucket item count to get to: 200000 curr count: 135695
2021/12/16 14:59:30 Waiting for src bucket item count to get to: 200000 curr count: 135695
2021/12/16 14:59:35 Waiting for src bucket item count to get to: 200000 curr count: 152232
2021/12/16 14:59:40 Waiting for src bucket item count to get to: 200000 curr count: 152232
2021/12/16 14:59:45 Waiting for src bucket item count to get to: 200000 curr count: 168776
2021/12/16 14:59:50 Waiting for src bucket item count to get to: 200000 curr count: 168776
2021/12/16 14:59:55 Waiting for src bucket item count to get to: 200000 curr count: 184253
2021/12/16 15:00:00 Waiting for src bucket item count to get to: 200000 curr count: 184253
2021/12/16 15:00:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 15:00:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 15:00:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:00:10 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 15:00:15 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 15:00:20 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 15:00:25 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 15:00:25 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/16 15:00:25 No eventing-con process running
2021/12/16 15:00:25 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/16 15:00:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:00:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 15:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:00:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:00:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:00:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 15:00:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (224.03s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/16 15:00:59 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 15:00:59 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/16 15:00:59 No eventing-con process running
2021/12/16 15:00:59 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/16 15:01:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:01:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:01:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:01:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:01:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:01:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:01:17 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:01:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 15:01:24 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 15:01:29 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 15:01:34 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 15:01:39 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 15:01:39 App: TestDuplicateWithKVSwapReb got deployed
2021/12/16 15:01:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 15:01:39 Starting bucket ops to source bucket
2021/12/16 15:01:39 Removing node(s): [] from the cluster
2021/12/16 15:01:44 Rebalance progress: 16.66666666666667
2021/12/16 15:01:49 Rebalance progress: 16.66666666666667
2021/12/16 15:01:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:01:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 15:01:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 15:01:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:01:59 Rebalance progress: 16.66666666666667
2021/12/16 15:02:04 Rebalance progress: 16.66666666666667
2021/12/16 15:02:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:02:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:02:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:02:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 15:02:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:02:14 Rebalance progress: 16.66666666666667
2021/12/16 15:02:19 Rebalance progress: 16.66666666666667
2021/12/16 15:02:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:02:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:02:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:02:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 15:02:24 Waiting for src bucket item count to get to: 200000 curr count: 63335
2021/12/16 15:02:29 Waiting for src bucket item count to get to: 200000 curr count: 63335
2021/12/16 15:02:34 Waiting for src bucket item count to get to: 200000 curr count: 78683
2021/12/16 15:02:39 Waiting for src bucket item count to get to: 200000 curr count: 78683
2021/12/16 15:02:44 Waiting for src bucket item count to get to: 200000 curr count: 94877
2021/12/16 15:02:49 Waiting for src bucket item count to get to: 200000 curr count: 94877
2021/12/16 15:02:55 Waiting for src bucket item count to get to: 200000 curr count: 111149
2021/12/16 15:03:00 Waiting for src bucket item count to get to: 200000 curr count: 111149
2021/12/16 15:03:05 Waiting for src bucket item count to get to: 200000 curr count: 127457
2021/12/16 15:03:10 Waiting for src bucket item count to get to: 200000 curr count: 127457
2021/12/16 15:03:15 Waiting for src bucket item count to get to: 200000 curr count: 143725
2021/12/16 15:03:20 Waiting for src bucket item count to get to: 200000 curr count: 143725
2021/12/16 15:03:25 Waiting for src bucket item count to get to: 200000 curr count: 159848
2021/12/16 15:03:30 Waiting for src bucket item count to get to: 200000 curr count: 159848
2021/12/16 15:03:35 Waiting for src bucket item count to get to: 200000 curr count: 175683
2021/12/16 15:03:40 Waiting for src bucket item count to get to: 200000 curr count: 175683
2021/12/16 15:03:45 Waiting for src bucket item count to get to: 200000 curr count: 191710
2021/12/16 15:03:50 Waiting for src bucket item count to get to: 200000 curr count: 191710
2021/12/16 15:03:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 15:03:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 15:03:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:04:00 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 15:04:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 15:04:10 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 15:04:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 15:04:15 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/16 15:04:15 No eventing-con process running
2021/12/16 15:04:15 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/16 15:04:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:04:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 15:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:04:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:04:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:04:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 15:04:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.53s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/16 15:04:48 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:04:48 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 15:04:48 No eventing-con process running
2021/12/16 15:04:48 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 15:04:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:04:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:04:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:04:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:05:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:05:07 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:05:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 15:05:13 Starting bucket ops to source bucket
2021/12/16 15:05:18 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:05:23 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:05:28 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:05:28 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:05:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 15:05:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:05:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:05:33 Rebalance progress: 16.66666666666667
2021/12/16 15:05:38 Rebalance progress: 16.66666666666667
2021/12/16 15:05:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:05:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 15:05:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:05:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:05:49 Rebalance progress: 16.66666666666667
2021/12/16 15:05:54 Rebalance progress: 16.66666666666667
2021/12/16 15:05:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:05:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 15:05:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 15:06:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:06:09 Rebalance progress: 12.5
2021/12/16 15:06:14 Rebalance progress: 12.5
2021/12/16 15:06:19 Rebalance progress: 12.5
2021/12/16 15:06:24 Rebalance progress: 90.7520325203252
2021/12/16 15:06:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:06:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:06:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:06:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:06:34 Rebalance progress: 12.5
2021/12/16 15:06:39 Rebalance progress: 12.5
2021/12/16 15:06:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:06:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:06:49 Rebalance progress: 12.5
2021/12/16 15:06:54 Rebalance progress: 12.5
2021/12/16 15:06:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:06:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:06:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:06:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 15:07:05 Rebalance progress: 12.5
2021/12/16 15:07:10 Rebalance progress: 12.5
2021/12/16 15:07:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:07:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:07:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:07:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:07:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:07:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:07:35 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 15:07:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:07:35 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 15:07:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:07:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:07:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12145
2021/12/16 15:07:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:07:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:07:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:07:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12145
2021/12/16 15:07:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.02s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/16 15:08:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 15:08:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/16 15:08:08 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:08:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/16 15:08:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:08:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:08:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:08:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:08:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:08:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:08:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:08:27 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:08:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/16 15:08:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:08:38 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 15:08:43 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 15:08:48 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 15:08:48 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:08:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:08:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:08:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:08:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:08:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 15:08:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:08:54 Rebalance progress: 12.5
2021/12/16 15:08:59 Rebalance progress: 12.5
2021/12/16 15:09:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:09:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:09:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:09:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 15:09:09 Rebalance progress: 12.5
2021/12/16 15:09:14 Rebalance progress: 12.5
2021/12/16 15:09:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:09:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:09:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:09:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:09:24 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 15:09:29 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 15:09:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 15:09:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 15:09:39 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/16 15:09:39 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:09:39 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/16 15:09:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:09:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:09:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4424
2021/12/16 15:09:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4424
2021/12/16 15:09:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:09:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:10:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:10:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4577
2021/12/16 15:10:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.38s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/16 15:10:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 15:10:18 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 15:10:18 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:10:18 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/16 15:10:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:10:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:10:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:10:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:10:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:10:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:10:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:10:36 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:10:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 15:10:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:10:47 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 15:10:52 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 15:10:57 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 15:10:57 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:10:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:10:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:10:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:10:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:11:03 Rebalance progress: 12.5
2021/12/16 15:11:08 Rebalance progress: 12.5
2021/12/16 15:11:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:11:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:11:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:11:18 Rebalance progress: 12.5
2021/12/16 15:11:23 Rebalance progress: 12.5
2021/12/16 15:11:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:11:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 15:11:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:11:33 Rebalance progress: 12.5
2021/12/16 15:11:38 Rebalance progress: 12.5
2021/12/16 15:11:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:11:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:11:48 Rebalance progress: 12.5
2021/12/16 15:11:53 Rebalance progress: 12.5
2021/12/16 15:11:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:11:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:11:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:12:04 Rebalance progress: 12.5
2021/12/16 15:12:09 Rebalance progress: 12.5
2021/12/16 15:12:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:12:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:12:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:12:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 15:12:19 Rebalance progress: 12.5
2021/12/16 15:12:24 Rebalance progress: 12.5
2021/12/16 15:12:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:12:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:12:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:12:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:12:34 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 15:12:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 15:12:44 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 15:12:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 15:12:49 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 15:12:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:12:49 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/16 15:12:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:12:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:13:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7726
2021/12/16 15:13:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7726
2021/12/16 15:13:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:13:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:13:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:13:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10604
2021/12/16 15:13:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (189.75s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/16 15:13:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 15:13:28 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/16 15:13:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:13:28 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/16 15:13:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:13:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:13:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:13:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:13:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:13:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:13:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:13:41 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:13:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 15:13:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:13:52 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 15:13:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 15:14:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 15:14:02 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:14:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:14:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:14:08 Rebalance progress: 12.5
2021/12/16 15:14:13 Rebalance progress: 12.5
2021/12/16 15:14:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:14:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:14:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:14:23 Rebalance progress: 12.5
2021/12/16 15:14:28 Rebalance progress: 12.5
2021/12/16 15:14:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:14:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:14:38 Rebalance progress: 12.5
2021/12/16 15:14:43 Rebalance progress: 12.5
2021/12/16 15:14:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:14:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:14:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:14:54 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 15:14:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 15:15:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 15:15:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 15:15:09 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/16 15:15:09 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:15:09 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/16 15:15:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:15:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:15:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3354
2021/12/16 15:15:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3354
2021/12/16 15:15:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:15:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:15:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:15:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6125
2021/12/16 15:15:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.78s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/16 15:15:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:15:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:15:48 Rebalance progress: 100
2021/12/16 15:15:53 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 15:15:53 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/16 15:15:53 No eventing-con process running
2021/12/16 15:15:53 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/16 15:15:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:16:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:16:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:16:06 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:16:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 15:16:12 Starting bucket ops to source bucket
2021/12/16 15:16:17 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 15:16:22 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 15:16:28 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 15:16:28 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:16:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:16:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:16:33 Rebalance progress: 12.5
2021/12/16 15:16:38 Rebalance progress: 12.5
2021/12/16 15:16:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:16:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 15:16:43 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 15:16:48 Rebalance progress: 12.5
2021/12/16 15:16:53 Rebalance progress: 12.5
2021/12/16 15:16:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:16:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:16:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:17:03 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 15:17:08 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 15:17:13 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 15:17:18 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 15:17:18 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/16 15:17:18 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:17:18 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/16 15:17:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:17:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:17:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 878
2021/12/16 15:17:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 878
2021/12/16 15:17:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:17:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:17:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:17:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4549
2021/12/16 15:17:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.60s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/16 15:17:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:17:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:17:58 Rebalance progress: 100
2021/12/16 15:18:03 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 15:18:03 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/16 15:18:03 No eventing-con process running
2021/12/16 15:18:03 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/16 15:18:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:18:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:18:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:18:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:18:16 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:18:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 15:18:22 Starting bucket ops to source bucket
2021/12/16 15:18:27 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 15:18:32 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 15:18:37 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 15:18:37 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:18:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:18:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:18:43 Rebalance progress: 12.5
2021/12/16 15:18:48 Rebalance progress: 12.5
2021/12/16 15:18:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:18:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:18:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 15:18:53 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 15:18:58 Rebalance progress: 12.5
2021/12/16 15:19:03 Rebalance progress: 12.5
2021/12/16 15:19:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:19:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:19:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:19:08 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 15:19:13 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 15:19:18 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 15:19:23 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 15:19:28 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 15:19:28 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/16 15:19:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:19:28 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/16 15:19:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:19:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:19:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 911
2021/12/16 15:19:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 911
2021/12/16 15:19:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:19:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:19:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:19:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37767
2021/12/16 15:19:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.84s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/16 15:20:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 15:20:07 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/16 15:20:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:20:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/16 15:20:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:20:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:20:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:20:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:20:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:20:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:20:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:20:25 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:20:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:20:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:20:26 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:20:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 15:20:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:20:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 15:20:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 15:20:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 15:20:48 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/16 15:20:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 15:20:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 15:21:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 15:21:03 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:21:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:21:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:21:08 Rebalance progress: 12.5
2021/12/16 15:21:13 Rebalance progress: 12.5
2021/12/16 15:21:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:21:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:21:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:21:24 Rebalance progress: 12.5
2021/12/16 15:21:29 Rebalance progress: 12.5
2021/12/16 15:21:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:21:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 15:21:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:21:39 Rebalance progress: 12.5
2021/12/16 15:21:44 Rebalance progress: 12.5
2021/12/16 15:21:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:21:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:21:54 Rebalance progress: 12.5
2021/12/16 15:21:59 Rebalance progress: 12.5
2021/12/16 15:22:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:22:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:22:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:22:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:22:09 Rebalance progress: 12.5
2021/12/16 15:22:14 Rebalance progress: 12.5
2021/12/16 15:22:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:22:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 15:22:25 Rebalance progress: 12.5
2021/12/16 15:22:30 Rebalance progress: 12.5
2021/12/16 15:22:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:22:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:22:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:22:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:22:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 15:22:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 15:22:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 15:22:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 15:22:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/16 15:22:55 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:22:55 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/16 15:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:23:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:23:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12189
2021/12/16 15:23:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:23:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:23:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:23:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1087
2021/12/16 15:23:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1087
2021/12/16 15:23:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:23:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:23:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 15:23:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 15:23:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 15:23:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 15:23:45 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/16 15:23:45 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:23:45 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/16 15:23:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:23:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:23:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:23:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:23:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:23:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (242.06s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/16 15:24:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 15:24:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/16 15:24:09 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:24:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/16 15:24:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:24:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:24:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:24:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:24:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:24:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:24:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:24:27 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:24:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:24:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:24:29 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:24:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 15:24:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:24:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 15:24:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 15:24:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 15:24:50 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/16 15:24:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 15:25:00 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 15:25:05 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 15:25:05 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:25:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:25:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:25:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:25:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:25:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 15:25:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:25:11 Rebalance progress: 12.5
2021/12/16 15:25:16 Rebalance progress: 12.5
2021/12/16 15:25:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:25:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:25:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:25:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 15:25:26 Rebalance progress: 12.5
2021/12/16 15:25:31 Rebalance progress: 12.5
2021/12/16 15:25:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:25:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:25:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:25:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 15:25:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 15:25:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 15:25:56 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 15:25:56 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/16 15:25:56 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:25:56 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/16 15:26:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:26:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:26:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:26:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:26:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:26:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6100
2021/12/16 15:26:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:26:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:26:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 15:26:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 15:26:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 15:26:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 15:26:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/16 15:26:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:26:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/16 15:26:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:26:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:26:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:26:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:26:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:26:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (171.72s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/16 15:27:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 15:27:01 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/16 15:27:01 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:27:01 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/16 15:27:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:27:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:27:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:27:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:27:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:27:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:27:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 15:27:19 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/16 15:27:19 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:27:19 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/16 15:27:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:27:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:27:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:27:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:27:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:27:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:27:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:27:33 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:27:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:27:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:27:34 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:27:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 15:27:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:27:45 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 15:27:50 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 15:27:55 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 15:27:55 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/16 15:28:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 15:28:05 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 15:28:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 15:28:10 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:28:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:28:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:28:15 Rebalance progress: 12.5
2021/12/16 15:28:20 Rebalance progress: 12.5
2021/12/16 15:28:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:28:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:28:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:28:31 Rebalance progress: 12.5
2021/12/16 15:28:36 Rebalance progress: 12.5
2021/12/16 15:28:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:28:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:28:46 Rebalance progress: 12.5
2021/12/16 15:28:51 Rebalance progress: 12.5
2021/12/16 15:28:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:28:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:29:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 15:29:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 15:29:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 15:29:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 15:29:16 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/16 15:29:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:29:16 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/16 15:29:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:29:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:29:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:29:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:29:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:29:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7585
2021/12/16 15:29:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:29:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:29:42 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 15:29:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 15:29:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 15:29:57 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 15:29:57 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/16 15:29:57 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:29:57 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/16 15:29:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:30:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:30:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:30:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:30:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:30:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (200.23s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 15:30:21 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:30:21 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 15:30:21 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:30:21 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 15:30:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:30:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:30:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:30:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:30:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:30:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:30:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:30:39 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:30:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 15:30:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:30:50 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:30:55 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:31:00 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:31:00 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:31:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:31:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:31:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:31:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:31:21 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:31:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:31:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:31:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:31:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:31:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:31:57 Rebalance progress: 12.5
2021/12/16 15:32:02 Rebalance progress: 12.5
2021/12/16 15:32:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:32:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:32:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:32:12 Rebalance progress: 12.5
2021/12/16 15:32:17 Rebalance progress: 12.5
2021/12/16 15:32:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:32:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:32:28 Rebalance progress: 12.5
2021/12/16 15:32:33 Rebalance progress: 12.5
2021/12/16 15:32:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:32:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:32:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:32:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:32:43 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:32:48 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:32:53 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:32:58 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:32:58 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 15:32:58 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:32:58 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 15:33:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:33:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:33:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:33:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:33:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:33:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11205
2021/12/16 15:33:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.57s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 15:33:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:33:26 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 15:33:26 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:33:26 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 15:33:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:33:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:33:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:33:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:33:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:33:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:33:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:33:45 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:33:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 15:33:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:33:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:34:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:34:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 15:34:06 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:34:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:34:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:34:26 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:34:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:34:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:34:57 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:34:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:35:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:35:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:35:28 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:35:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:35:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:35:59 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:35:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:36:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:36:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:36:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:36:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:36:45 Rebalance progress: 12.5
2021/12/16 15:36:50 Rebalance progress: 12.5
2021/12/16 15:36:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:36:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:36:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:36:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:37:01 Rebalance progress: 12.5
2021/12/16 15:37:06 Rebalance progress: 12.5
2021/12/16 15:37:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:37:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:37:16 Rebalance progress: 12.5
2021/12/16 15:37:21 Rebalance progress: 12.5
2021/12/16 15:37:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:37:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:37:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:37:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:37:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:37:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:37:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:37:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 15:37:46 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 15:37:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:37:46 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 15:37:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:37:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:37:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21441
2021/12/16 15:37:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:37:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:38:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:38:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (288.15s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/16 15:38:14 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 15:38:14 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/16 15:38:14 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:38:14 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/16 15:38:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:38:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:38:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:38:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:38:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:38:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:38:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:38:33 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:38:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 15:38:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:38:44 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 15:38:49 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 15:38:54 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 15:38:54 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:38:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:38:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:38:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:38:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:39:00 Rebalance progress: 12.5
2021/12/16 15:39:05 Rebalance progress: 12.5
2021/12/16 15:39:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:39:10 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:39:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:39:25 Rebalance progress: 12.5
2021/12/16 15:39:30 Rebalance progress: 12.5
2021/12/16 15:39:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:39:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:39:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:39:41 Rebalance progress: 12.5
2021/12/16 15:39:46 Rebalance progress: 12.5
2021/12/16 15:39:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:39:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:39:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:39:56 Rebalance progress: 12.5
2021/12/16 15:40:01 Rebalance progress: 12.5
2021/12/16 15:40:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:40:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:40:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:40:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:40:11 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 15:40:16 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 15:40:21 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 15:40:26 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 15:40:26 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/16 15:40:26 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:40:26 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/16 15:40:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:40:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:40:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8665
2021/12/16 15:40:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:40:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:40:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:40:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (160.16s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/16 15:40:55 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 15:40:55 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/16 15:40:55 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:40:55 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/16 15:40:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:41:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:41:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:41:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:41:13 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:41:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 15:41:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:41:24 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 15:41:29 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 15:41:34 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 15:41:34 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:41:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/16 15:41:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:41:40 Rebalance progress: 12.5
2021/12/16 15:41:45 Rebalance progress: 12.5
2021/12/16 15:41:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:41:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:41:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/16 15:41:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:41:55 Rebalance progress: 12.5
2021/12/16 15:42:00 Rebalance progress: 12.5
2021/12/16 15:42:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:42:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:42:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:42:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:42:10 Rebalance progress: 12.5
2021/12/16 15:42:15 Rebalance progress: 12.5
2021/12/16 15:42:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:42:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:42:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:42:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:42:25 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 15:42:31 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 15:42:36 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 15:42:41 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 15:42:41 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/16 15:42:41 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:42:41 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/16 15:42:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:42:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:42:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6092
2021/12/16 15:42:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6092
2021/12/16 15:42:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:42:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:43:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:43:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6092
2021/12/16 15:43:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (144.54s)
=== RUN   TestEventingRebInWithOps
2021/12/16 15:43:19 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 15:43:19 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/16 15:43:19 eventingSetup.go:46: No eventing-con process running
2021/12/16 15:43:19 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/16 15:43:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:43:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:43:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:43:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:43:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:43:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:43:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:43:38 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:43:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 15:43:44 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 15:43:49 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 15:43:54 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 15:43:59 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 15:43:59 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/16 15:43:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:43:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 15:43:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 15:44:05 Rebalance progress: 12.5
2021/12/16 15:44:10 Rebalance progress: 12.5
2021/12/16 15:44:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:44:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:44:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:44:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:44:20 Rebalance progress: 12.5
2021/12/16 15:44:25 Rebalance progress: 12.5
2021/12/16 15:44:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:44:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:44:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:44:30 Waiting for src bucket item count to get to: 200000 curr count: 30679
2021/12/16 15:44:35 Waiting for src bucket item count to get to: 200000 curr count: 46615
2021/12/16 15:44:40 Waiting for src bucket item count to get to: 200000 curr count: 46615
2021/12/16 15:44:45 Waiting for src bucket item count to get to: 200000 curr count: 62645
2021/12/16 15:44:50 Waiting for src bucket item count to get to: 200000 curr count: 62645
2021/12/16 15:44:55 Waiting for src bucket item count to get to: 200000 curr count: 77696
2021/12/16 15:45:00 Waiting for src bucket item count to get to: 200000 curr count: 77696
2021/12/16 15:45:05 Waiting for src bucket item count to get to: 200000 curr count: 93215
2021/12/16 15:45:10 Waiting for src bucket item count to get to: 200000 curr count: 93215
2021/12/16 15:45:15 Waiting for src bucket item count to get to: 200000 curr count: 109469
2021/12/16 15:45:20 Waiting for src bucket item count to get to: 200000 curr count: 109469
2021/12/16 15:45:25 Waiting for src bucket item count to get to: 200000 curr count: 125620
2021/12/16 15:45:30 Waiting for src bucket item count to get to: 200000 curr count: 141774
2021/12/16 15:45:35 Waiting for src bucket item count to get to: 200000 curr count: 141774
2021/12/16 15:45:40 Waiting for src bucket item count to get to: 200000 curr count: 141774
2021/12/16 15:45:45 Waiting for src bucket item count to get to: 200000 curr count: 158049
2021/12/16 15:45:50 Waiting for src bucket item count to get to: 200000 curr count: 158049
2021/12/16 15:45:55 Waiting for src bucket item count to get to: 200000 curr count: 173355
2021/12/16 15:46:00 Waiting for src bucket item count to get to: 200000 curr count: 173355
2021/12/16 15:46:05 Waiting for src bucket item count to get to: 200000 curr count: 189155
2021/12/16 15:46:10 Waiting for src bucket item count to get to: 200000 curr count: 189155
2021/12/16 15:46:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 15:46:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 15:46:15 Starting bucket ops to source bucket
2021/12/16 15:50:05 Waiting for dst bucket item count to get to: 0 curr count: 19032
2021/12/16 15:50:10 Waiting for dst bucket item count to get to: 0 curr count: 884
2021/12/16 15:50:15 Waiting for dst bucket item count to get to: 0 curr count: 884
2021/12/16 15:50:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:50:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 15:50:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:50:20 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 15:50:25 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 15:50:30 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 15:50:35 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 15:50:35 App: TestEventingRebInWithOps got un-deployed
2021/12/16 15:50:35 No eventing-con process running
2021/12/16 15:50:35 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/16 15:50:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:50:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:50:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:50:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:50:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:50:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.42s)
=== RUN   TestEventingRebOutWithOps
2021/12/16 15:50:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:50:54 Removing node(s): [] from the cluster
2021/12/16 15:50:59 Rebalance progress: 100
2021/12/16 15:51:04 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 15:51:04 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 15:51:04 No eventing-con process running
2021/12/16 15:51:04 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/16 15:51:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:51:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:51:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:51:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:51:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:51:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:51:23 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:51:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 15:51:29 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 15:51:34 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 15:51:39 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 15:51:44 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 15:51:44 App: TestEventingRebOutWithOps got deployed
2021/12/16 15:51:44 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:51:44 Starting bucket ops to source bucket
2021/12/16 15:51:49 Rebalance progress: 12.5
2021/12/16 15:51:54 Rebalance progress: 12.5
2021/12/16 15:51:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:51:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:51:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:51:59 Waiting for src bucket item count to get to: 200000 curr count: 7913
2021/12/16 15:52:04 Waiting for src bucket item count to get to: 200000 curr count: 23917
2021/12/16 15:52:09 Waiting for src bucket item count to get to: 200000 curr count: 23917
2021/12/16 15:52:14 Waiting for src bucket item count to get to: 200000 curr count: 40103
2021/12/16 15:52:19 Waiting for src bucket item count to get to: 200000 curr count: 40103
2021/12/16 15:52:24 Waiting for src bucket item count to get to: 200000 curr count: 56217
2021/12/16 15:52:29 Waiting for src bucket item count to get to: 200000 curr count: 56217
2021/12/16 15:52:34 Waiting for src bucket item count to get to: 200000 curr count: 56217
2021/12/16 15:52:39 Waiting for src bucket item count to get to: 200000 curr count: 72381
2021/12/16 15:52:44 Waiting for src bucket item count to get to: 200000 curr count: 72381
2021/12/16 15:52:49 Waiting for src bucket item count to get to: 200000 curr count: 87161
2021/12/16 15:52:54 Waiting for src bucket item count to get to: 200000 curr count: 87161
2021/12/16 15:52:59 Waiting for src bucket item count to get to: 200000 curr count: 103323
2021/12/16 15:53:04 Waiting for src bucket item count to get to: 200000 curr count: 103323
2021/12/16 15:53:09 Waiting for src bucket item count to get to: 200000 curr count: 119491
2021/12/16 15:53:14 Waiting for src bucket item count to get to: 200000 curr count: 135676
2021/12/16 15:53:19 Waiting for src bucket item count to get to: 200000 curr count: 135676
2021/12/16 15:53:24 Waiting for src bucket item count to get to: 200000 curr count: 151564
2021/12/16 15:53:29 Waiting for src bucket item count to get to: 200000 curr count: 151564
2021/12/16 15:53:34 Waiting for src bucket item count to get to: 200000 curr count: 167813
2021/12/16 15:53:39 Waiting for src bucket item count to get to: 200000 curr count: 167813
2021/12/16 15:53:44 Waiting for src bucket item count to get to: 200000 curr count: 182719
2021/12/16 15:53:49 Waiting for src bucket item count to get to: 200000 curr count: 182719
2021/12/16 15:53:54 Waiting for src bucket item count to get to: 200000 curr count: 198930
2021/12/16 15:53:59 Waiting for src bucket item count to get to: 200000 curr count: 198930
2021/12/16 15:54:04 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 15:54:04 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 15:54:04 Starting bucket ops to source bucket
2021/12/16 15:57:54 Waiting for dst bucket item count to get to: 0 curr count: 17701
2021/12/16 15:57:59 Waiting for dst bucket item count to get to: 0 curr count: 17701
2021/12/16 15:57:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:57:59 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 15:57:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 15:58:04 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 15:58:09 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 15:58:14 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 15:58:19 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 15:58:19 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 15:58:19 No eventing-con process running
2021/12/16 15:58:19 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/16 15:58:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:58:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:58:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:58:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:58:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (464.20s)
=== RUN   TestEventingSwapRebWithOps
2021/12/16 15:58:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 15:58:43 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 15:58:43 No eventing-con process running
2021/12/16 15:58:43 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/16 15:58:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:58:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:58:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 15:58:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:58:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:58:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:59:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 15:59:01 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 15:59:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 15:59:08 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 15:59:13 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 15:59:18 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 15:59:23 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 15:59:23 App: TestEventingSwapRebWithOps got deployed
2021/12/16 15:59:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 15:59:23 Starting bucket ops to source bucket
2021/12/16 15:59:23 Removing node(s): [] from the cluster
2021/12/16 15:59:28 Rebalance progress: 12.5
2021/12/16 15:59:33 Rebalance progress: 12.5
2021/12/16 15:59:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:59:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:59:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:59:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 15:59:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 15:59:44 Rebalance progress: 12.5
2021/12/16 15:59:49 Rebalance progress: 12.5
2021/12/16 15:59:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:59:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 15:59:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 15:59:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 15:59:59 Rebalance progress: 12.5
2021/12/16 16:00:04 Rebalance progress: 12.5
2021/12/16 16:00:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:00:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:00:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:00:09 Waiting for src bucket item count to get to: 200000 curr count: 55660
2021/12/16 16:00:14 Waiting for src bucket item count to get to: 200000 curr count: 70611
2021/12/16 16:00:19 Waiting for src bucket item count to get to: 200000 curr count: 70611
2021/12/16 16:00:24 Waiting for src bucket item count to get to: 200000 curr count: 86185
2021/12/16 16:00:29 Waiting for src bucket item count to get to: 200000 curr count: 86185
2021/12/16 16:00:34 Waiting for src bucket item count to get to: 200000 curr count: 102471
2021/12/16 16:00:39 Waiting for src bucket item count to get to: 200000 curr count: 102471
2021/12/16 16:00:44 Waiting for src bucket item count to get to: 200000 curr count: 118788
2021/12/16 16:00:49 Waiting for src bucket item count to get to: 200000 curr count: 118788
2021/12/16 16:00:54 Waiting for src bucket item count to get to: 200000 curr count: 134948
2021/12/16 16:00:59 Waiting for src bucket item count to get to: 200000 curr count: 134948
2021/12/16 16:01:04 Waiting for src bucket item count to get to: 200000 curr count: 151225
2021/12/16 16:01:09 Waiting for src bucket item count to get to: 200000 curr count: 151225
2021/12/16 16:01:14 Waiting for src bucket item count to get to: 200000 curr count: 166683
2021/12/16 16:01:19 Waiting for src bucket item count to get to: 200000 curr count: 166683
2021/12/16 16:01:24 Waiting for src bucket item count to get to: 200000 curr count: 182197
2021/12/16 16:01:29 Waiting for src bucket item count to get to: 200000 curr count: 182197
2021/12/16 16:01:34 Waiting for src bucket item count to get to: 200000 curr count: 198405
2021/12/16 16:01:39 Waiting for src bucket item count to get to: 200000 curr count: 198405
2021/12/16 16:01:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 16:01:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 16:01:44 Starting bucket ops to source bucket
2021/12/16 16:05:34 Waiting for dst bucket item count to get to: 0 curr count: 17739
2021/12/16 16:05:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:05:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:05:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:05:39 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 16:05:44 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 16:05:49 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 16:05:54 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 16:05:54 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 16:05:54 No eventing-con process running
2021/12/16 16:05:54 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/16 16:05:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:06:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:06:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:06:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:06:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:06:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (454.93s)
=== RUN   TestKVRebInWithOps
2021/12/16 16:06:18 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 16:06:18 App: TestKVRebInWithOps got un-deployed
2021/12/16 16:06:18 No eventing-con process running
2021/12/16 16:06:18 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/16 16:06:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:06:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:06:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:06:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:06:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:06:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:06:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:06:36 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:06:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 16:06:42 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 16:06:47 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 16:06:52 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 16:06:57 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 16:06:57 App: TestKVRebInWithOps got deployed
2021/12/16 16:06:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 16:06:57 Starting bucket ops to source bucket
2021/12/16 16:06:58 Removing node(s): [] from the cluster
2021/12/16 16:07:03 Rebalance progress: 12.5
2021/12/16 16:07:08 Rebalance progress: 12.5
2021/12/16 16:07:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:07:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:07:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:07:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:07:18 Rebalance progress: 12.5
2021/12/16 16:07:23 Rebalance progress: 12.5
2021/12/16 16:07:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:07:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:07:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:07:28 Waiting for src bucket item count to get to: 200000 curr count: 33198
2021/12/16 16:07:33 Waiting for src bucket item count to get to: 200000 curr count: 49191
2021/12/16 16:07:38 Waiting for src bucket item count to get to: 200000 curr count: 49191
2021/12/16 16:07:43 Waiting for src bucket item count to get to: 200000 curr count: 65493
2021/12/16 16:07:48 Waiting for src bucket item count to get to: 200000 curr count: 65493
2021/12/16 16:07:53 Waiting for src bucket item count to get to: 200000 curr count: 80351
2021/12/16 16:07:58 Waiting for src bucket item count to get to: 200000 curr count: 80351
2021/12/16 16:08:03 Waiting for src bucket item count to get to: 200000 curr count: 96433
2021/12/16 16:08:08 Waiting for src bucket item count to get to: 200000 curr count: 96433
2021/12/16 16:08:13 Waiting for src bucket item count to get to: 200000 curr count: 112586
2021/12/16 16:08:18 Waiting for src bucket item count to get to: 200000 curr count: 112586
2021/12/16 16:08:23 Waiting for src bucket item count to get to: 200000 curr count: 112586
2021/12/16 16:08:28 Waiting for src bucket item count to get to: 200000 curr count: 128751
2021/12/16 16:08:33 Waiting for src bucket item count to get to: 200000 curr count: 128751
2021/12/16 16:08:38 Waiting for src bucket item count to get to: 200000 curr count: 145165
2021/12/16 16:08:43 Waiting for src bucket item count to get to: 200000 curr count: 145165
2021/12/16 16:08:48 Waiting for src bucket item count to get to: 200000 curr count: 161213
2021/12/16 16:08:53 Waiting for src bucket item count to get to: 200000 curr count: 161213
2021/12/16 16:08:58 Waiting for src bucket item count to get to: 200000 curr count: 176120
2021/12/16 16:09:03 Waiting for src bucket item count to get to: 200000 curr count: 176120
2021/12/16 16:09:08 Waiting for src bucket item count to get to: 200000 curr count: 192163
2021/12/16 16:09:13 Waiting for src bucket item count to get to: 200000 curr count: 192163
2021/12/16 16:09:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 16:09:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 16:09:18 Starting bucket ops to source bucket
2021/12/16 16:13:08 Waiting for dst bucket item count to get to: 0 curr count: 7354
2021/12/16 16:13:13 Waiting for dst bucket item count to get to: 0 curr count: 7354
2021/12/16 16:13:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:13:13 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:13:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:13:18 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 16:13:23 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 16:13:29 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 16:13:34 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 16:13:34 App: TestKVRebInWithOps got un-deployed
2021/12/16 16:13:34 No eventing-con process running
2021/12/16 16:13:34 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/16 16:13:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:13:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:13:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:13:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:13:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.41s)
=== RUN   TestKVRebOutWithOps
2021/12/16 16:13:52 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 16:13:53 Removing node(s): [] from the cluster
2021/12/16 16:13:58 Rebalance progress: 100
2021/12/16 16:14:03 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 16:14:03 App: TestKVRebOutWithOps got un-deployed
2021/12/16 16:14:03 No eventing-con process running
2021/12/16 16:14:03 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/16 16:14:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:14:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:14:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:14:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:14:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:14:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:14:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:14:21 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:14:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 16:14:27 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 16:14:32 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 16:14:37 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 16:14:42 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 16:14:42 App: TestKVRebOutWithOps got deployed
2021/12/16 16:14:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:14:42 Starting bucket ops to source bucket
2021/12/16 16:14:47 Rebalance progress: 12.5
2021/12/16 16:14:52 Rebalance progress: 12.5
2021/12/16 16:14:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:14:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:14:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:14:57 Waiting for src bucket item count to get to: 200000 curr count: 10009
2021/12/16 16:15:03 Waiting for src bucket item count to get to: 200000 curr count: 26020
2021/12/16 16:15:08 Waiting for src bucket item count to get to: 200000 curr count: 26020
2021/12/16 16:15:13 Waiting for src bucket item count to get to: 200000 curr count: 42001
2021/12/16 16:15:18 Waiting for src bucket item count to get to: 200000 curr count: 42001
2021/12/16 16:15:23 Waiting for src bucket item count to get to: 200000 curr count: 58189
2021/12/16 16:15:28 Waiting for src bucket item count to get to: 200000 curr count: 58189
2021/12/16 16:15:33 Waiting for src bucket item count to get to: 200000 curr count: 58189
2021/12/16 16:15:38 Waiting for src bucket item count to get to: 200000 curr count: 73508
2021/12/16 16:15:43 Waiting for src bucket item count to get to: 200000 curr count: 73508
2021/12/16 16:15:48 Waiting for src bucket item count to get to: 200000 curr count: 89511
2021/12/16 16:15:53 Waiting for src bucket item count to get to: 200000 curr count: 89511
2021/12/16 16:15:58 Waiting for src bucket item count to get to: 200000 curr count: 105483
2021/12/16 16:16:03 Waiting for src bucket item count to get to: 200000 curr count: 105483
2021/12/16 16:16:08 Waiting for src bucket item count to get to: 200000 curr count: 121324
2021/12/16 16:16:13 Waiting for src bucket item count to get to: 200000 curr count: 121324
2021/12/16 16:16:18 Waiting for src bucket item count to get to: 200000 curr count: 137243
2021/12/16 16:16:23 Waiting for src bucket item count to get to: 200000 curr count: 137243
2021/12/16 16:16:28 Waiting for src bucket item count to get to: 200000 curr count: 153460
2021/12/16 16:16:33 Waiting for src bucket item count to get to: 200000 curr count: 153460
2021/12/16 16:16:38 Waiting for src bucket item count to get to: 200000 curr count: 169053
2021/12/16 16:16:43 Waiting for src bucket item count to get to: 200000 curr count: 169053
2021/12/16 16:16:48 Waiting for src bucket item count to get to: 200000 curr count: 184624
2021/12/16 16:16:53 Waiting for src bucket item count to get to: 200000 curr count: 184624
2021/12/16 16:16:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 16:16:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 16:16:58 Starting bucket ops to source bucket
2021/12/16 16:20:48 Waiting for dst bucket item count to get to: 0 curr count: 6736
2021/12/16 16:20:53 Waiting for dst bucket item count to get to: 0 curr count: 6736
2021/12/16 16:20:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:20:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:20:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:20:58 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 16:21:03 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 16:21:08 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 16:21:13 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 16:21:13 App: TestKVRebOutWithOps got un-deployed
2021/12/16 16:21:13 No eventing-con process running
2021/12/16 16:21:13 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/16 16:21:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:21:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:21:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:21:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:21:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (459.67s)
=== RUN   TestKVSwapRebWithOps
2021/12/16 16:21:37 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 16:21:37 App: TestKVSwapRebWithOps got un-deployed
2021/12/16 16:21:37 No eventing-con process running
2021/12/16 16:21:37 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/16 16:21:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:21:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:21:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:21:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:21:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:21:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:21:55 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:21:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 16:22:01 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 16:22:06 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 16:22:11 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 16:22:16 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 16:22:16 App: TestKVSwapRebWithOps got deployed
2021/12/16 16:22:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 16:22:16 Starting bucket ops to source bucket
2021/12/16 16:22:17 Removing node(s): [] from the cluster
2021/12/16 16:22:22 Rebalance progress: 12.5
2021/12/16 16:22:27 Rebalance progress: 12.5
2021/12/16 16:22:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:22:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 16:22:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:22:38 Rebalance progress: 12.5
2021/12/16 16:22:43 Rebalance progress: 12.5
2021/12/16 16:22:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:22:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:22:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:22:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 16:22:53 Rebalance progress: 12.5
2021/12/16 16:22:58 Rebalance progress: 12.5
2021/12/16 16:23:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:23:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:23:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:23:03 Waiting for src bucket item count to get to: 200000 curr count: 65564
2021/12/16 16:23:08 Waiting for src bucket item count to get to: 200000 curr count: 65564
2021/12/16 16:23:13 Waiting for src bucket item count to get to: 200000 curr count: 80075
2021/12/16 16:23:18 Waiting for src bucket item count to get to: 200000 curr count: 80075
2021/12/16 16:23:23 Waiting for src bucket item count to get to: 200000 curr count: 96021
2021/12/16 16:23:28 Waiting for src bucket item count to get to: 200000 curr count: 96021
2021/12/16 16:23:33 Waiting for src bucket item count to get to: 200000 curr count: 111916
2021/12/16 16:23:38 Waiting for src bucket item count to get to: 200000 curr count: 111916
2021/12/16 16:23:43 Waiting for src bucket item count to get to: 200000 curr count: 127974
2021/12/16 16:23:48 Waiting for src bucket item count to get to: 200000 curr count: 127974
2021/12/16 16:23:53 Waiting for src bucket item count to get to: 200000 curr count: 144190
2021/12/16 16:23:58 Waiting for src bucket item count to get to: 200000 curr count: 144190
2021/12/16 16:24:03 Waiting for src bucket item count to get to: 200000 curr count: 160368
2021/12/16 16:24:08 Waiting for src bucket item count to get to: 200000 curr count: 160368
2021/12/16 16:24:13 Waiting for src bucket item count to get to: 200000 curr count: 175182
2021/12/16 16:24:18 Waiting for src bucket item count to get to: 200000 curr count: 175182
2021/12/16 16:24:23 Waiting for src bucket item count to get to: 200000 curr count: 191530
2021/12/16 16:24:28 Waiting for src bucket item count to get to: 200000 curr count: 191530
2021/12/16 16:24:33 Waiting for src bucket item count to get to: 200000 curr count: 191530
2021/12/16 16:24:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 16:24:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 16:24:38 Starting bucket ops to source bucket
2021/12/16 16:28:28 Waiting for dst bucket item count to get to: 0 curr count: 5839
2021/12/16 16:28:33 Waiting for dst bucket item count to get to: 0 curr count: 5839
2021/12/16 16:28:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:28:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:28:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:28:38 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 16:28:43 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 16:28:48 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 16:28:53 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 16:28:53 App: TestKVSwapRebWithOps got un-deployed
2021/12/16 16:28:53 No eventing-con process running
2021/12/16 16:28:53 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/16 16:28:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:28:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:28:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:29:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:29:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (459.49s)
=== RUN   TestEventingRebInWithTimers
2021/12/16 16:29:16 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 16:29:16 App: TestEventingRebInWithTimers got un-deployed
2021/12/16 16:29:16 No eventing-con process running
2021/12/16 16:29:16 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/16 16:29:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:29:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:29:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:29:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:29:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:29:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:29:35 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:29:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 16:29:41 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 16:29:46 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 16:29:51 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 16:29:56 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 16:29:56 App: TestEventingRebInWithTimers got deployed
2021/12/16 16:29:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 16:29:56 Starting bucket ops to source bucket
2021/12/16 16:29:57 Removing node(s): [] from the cluster
2021/12/16 16:30:02 Rebalance progress: 12.5
2021/12/16 16:30:07 Rebalance progress: 12.5
2021/12/16 16:30:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:30:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:30:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:30:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:30:17 Rebalance progress: 12.5
2021/12/16 16:30:22 Rebalance progress: 12.5
2021/12/16 16:30:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:30:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:30:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:30:27 Waiting for src bucket item count to get to: 50000 curr count: 28166
2021/12/16 16:30:32 Waiting for src bucket item count to get to: 50000 curr count: 39396
2021/12/16 16:30:37 Waiting for src bucket item count to get to: 50000 curr count: 39396
2021/12/16 16:30:47 Waiting for dst bucket item count to get to: 50000 curr count: 36022
2021/12/16 16:30:52 Waiting for dst bucket item count to get to: 50000 curr count: 36022
2021/12/16 16:30:57 Waiting for dst bucket item count to get to: 50000 curr count: 45025
2021/12/16 16:31:02 Waiting for dst bucket item count to get to: 50000 curr count: 45025
2021/12/16 16:31:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:31:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:31:02 Starting bucket ops to source bucket
2021/12/16 16:32:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:32:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:32:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:32:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:32:32 Waiting for dst bucket item count to get to: 0 curr count: 41231
2021/12/16 16:32:37 Waiting for dst bucket item count to get to: 0 curr count: 41231
2021/12/16 16:32:42 Waiting for dst bucket item count to get to: 0 curr count: 31352
2021/12/16 16:32:47 Waiting for dst bucket item count to get to: 0 curr count: 31352
2021/12/16 16:32:52 Waiting for dst bucket item count to get to: 0 curr count: 6220
2021/12/16 16:32:57 Waiting for dst bucket item count to get to: 0 curr count: 6220
2021/12/16 16:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:32:57 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:32:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:33:02 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 16:33:07 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 16:33:12 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 16:33:17 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 16:33:17 App: TestEventingRebInWithTimers got un-deployed
2021/12/16 16:33:17 No eventing-con process running
2021/12/16 16:33:17 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/16 16:33:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:33:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:33:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:33:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:33:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (264.73s)
=== RUN   TestEventingRebOutWithTimers
2021/12/16 16:33:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 16:33:36 Removing node(s): [] from the cluster
2021/12/16 16:33:41 Rebalance progress: 100
2021/12/16 16:33:46 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 16:33:46 App: TestEventingRebOutWithTimers got un-deployed
2021/12/16 16:33:46 No eventing-con process running
2021/12/16 16:33:46 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/16 16:33:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:33:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:33:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:34:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:34:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:34:05 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:34:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 16:34:11 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 16:34:16 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 16:34:21 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 16:34:26 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 16:34:26 App: TestEventingRebOutWithTimers got deployed
2021/12/16 16:34:26 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:34:26 Starting bucket ops to source bucket
2021/12/16 16:34:31 Rebalance progress: 12.5
2021/12/16 16:34:36 Rebalance progress: 12.5
2021/12/16 16:34:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:34:41 Waiting for src bucket item count to get to: 50000 curr count: 18336
2021/12/16 16:34:46 Waiting for src bucket item count to get to: 50000 curr count: 18336
2021/12/16 16:34:51 Waiting for src bucket item count to get to: 50000 curr count: 27835
2021/12/16 16:34:56 Waiting for src bucket item count to get to: 50000 curr count: 27835
2021/12/16 16:35:01 Waiting for src bucket item count to get to: 50000 curr count: 39833
2021/12/16 16:35:06 Waiting for src bucket item count to get to: 50000 curr count: 39833
2021/12/16 16:35:11 Waiting for src bucket item count to get to: 50000 curr count: 39833
2021/12/16 16:35:21 Waiting for dst bucket item count to get to: 50000 curr count: 39615
2021/12/16 16:35:26 Waiting for dst bucket item count to get to: 50000 curr count: 39615
2021/12/16 16:35:31 Waiting for dst bucket item count to get to: 50000 curr count: 48980
2021/12/16 16:35:36 Waiting for dst bucket item count to get to: 50000 curr count: 48980
2021/12/16 16:35:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:35:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:35:36 Starting bucket ops to source bucket
2021/12/16 16:36:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:36:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:36:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:37:01 Waiting for dst bucket item count to get to: 0 curr count: 44520
2021/12/16 16:37:06 Waiting for dst bucket item count to get to: 0 curr count: 44520
2021/12/16 16:37:11 Waiting for dst bucket item count to get to: 0 curr count: 32158
2021/12/16 16:37:16 Waiting for dst bucket item count to get to: 0 curr count: 32158
2021/12/16 16:37:21 Waiting for dst bucket item count to get to: 0 curr count: 8361
2021/12/16 16:37:26 Waiting for dst bucket item count to get to: 0 curr count: 8361
2021/12/16 16:37:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:37:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:37:32 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 16:37:37 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 16:37:42 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 16:37:47 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 16:37:47 App: TestEventingRebOutWithTimers got un-deployed
2021/12/16 16:37:47 No eventing-con process running
2021/12/16 16:37:47 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/16 16:37:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:37:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:37:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:37:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.17s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/16 16:38:10 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 16:38:10 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/16 16:38:10 No eventing-con process running
2021/12/16 16:38:10 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/16 16:38:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:38:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:38:29 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:38:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 16:38:35 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 16:38:40 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 16:38:45 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 16:38:50 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 16:38:50 App: TestEventingSwapRebWithTimers got deployed
2021/12/16 16:38:50 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 16:38:50 Starting bucket ops to source bucket
2021/12/16 16:38:50 Removing node(s): [] from the cluster
2021/12/16 16:38:55 Rebalance progress: 12.5
2021/12/16 16:39:00 Rebalance progress: 12.5
2021/12/16 16:39:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:39:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:39:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 16:39:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:39:11 Rebalance progress: 12.5
2021/12/16 16:39:16 Rebalance progress: 12.5
2021/12/16 16:39:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:39:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 16:39:26 Rebalance progress: 12.5
2021/12/16 16:39:31 Rebalance progress: 12.5
2021/12/16 16:39:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:39:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:39:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:39:36 Waiting for src bucket item count to get to: 50000 curr count: 46567
2021/12/16 16:39:46 Waiting for dst bucket item count to get to: 50000 curr count: 38784
2021/12/16 16:39:51 Waiting for dst bucket item count to get to: 50000 curr count: 38784
2021/12/16 16:39:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:39:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:39:51 Starting bucket ops to source bucket
2021/12/16 16:41:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:41:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:41:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:41:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:41:21 Waiting for dst bucket item count to get to: 0 curr count: 43020
2021/12/16 16:41:26 Waiting for dst bucket item count to get to: 0 curr count: 43020
2021/12/16 16:41:31 Waiting for dst bucket item count to get to: 0 curr count: 23781
2021/12/16 16:41:36 Waiting for dst bucket item count to get to: 0 curr count: 23781
2021/12/16 16:41:41 Waiting for dst bucket item count to get to: 0 curr count: 12240
2021/12/16 16:41:46 Waiting for dst bucket item count to get to: 0 curr count: 12240
2021/12/16 16:41:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:41:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:41:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:41:51 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 16:41:56 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 16:42:01 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 16:42:06 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 16:42:06 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/16 16:42:06 No eventing-con process running
2021/12/16 16:42:06 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/16 16:42:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:42:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:42:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:42:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:42:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:42:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (259.42s)
=== RUN   TestKVRebInWithTimers
2021/12/16 16:42:30 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 16:42:30 App: TestKVRebInWithTimers got un-deployed
2021/12/16 16:42:30 No eventing-con process running
2021/12/16 16:42:30 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/16 16:42:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:42:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:42:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:42:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:42:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:42:48 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:42:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 16:42:54 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 16:42:59 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 16:43:04 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 16:43:09 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 16:43:09 App: TestKVRebInWithTimers got deployed
2021/12/16 16:43:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 16:43:09 Starting bucket ops to source bucket
2021/12/16 16:43:10 Removing node(s): [] from the cluster
2021/12/16 16:43:15 Rebalance progress: 12.5
2021/12/16 16:43:20 Rebalance progress: 12.5
2021/12/16 16:43:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:43:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:43:30 Rebalance progress: 12.5
2021/12/16 16:43:35 Rebalance progress: 12.5
2021/12/16 16:43:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:43:40 Waiting for src bucket item count to get to: 50000 curr count: 26651
2021/12/16 16:43:45 Waiting for src bucket item count to get to: 50000 curr count: 36519
2021/12/16 16:43:50 Waiting for src bucket item count to get to: 50000 curr count: 36519
2021/12/16 16:43:55 Waiting for src bucket item count to get to: 50000 curr count: 48122
2021/12/16 16:44:00 Waiting for src bucket item count to get to: 50000 curr count: 48122
2021/12/16 16:44:10 Waiting for dst bucket item count to get to: 50000 curr count: 41143
2021/12/16 16:44:15 Waiting for dst bucket item count to get to: 50000 curr count: 49992
2021/12/16 16:44:20 Waiting for dst bucket item count to get to: 50000 curr count: 49992
2021/12/16 16:44:20 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:44:20 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:44:20 Starting bucket ops to source bucket
2021/12/16 16:45:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:45:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:45:41 Waiting for dst bucket item count to get to: 0 curr count: 49185
2021/12/16 16:45:46 Waiting for dst bucket item count to get to: 0 curr count: 49185
2021/12/16 16:45:51 Waiting for dst bucket item count to get to: 0 curr count: 39502
2021/12/16 16:45:56 Waiting for dst bucket item count to get to: 0 curr count: 39502
2021/12/16 16:46:01 Waiting for dst bucket item count to get to: 0 curr count: 29673
2021/12/16 16:46:06 Waiting for dst bucket item count to get to: 0 curr count: 29673
2021/12/16 16:46:11 Waiting for dst bucket item count to get to: 0 curr count: 8322
2021/12/16 16:46:16 Waiting for dst bucket item count to get to: 0 curr count: 8322
2021/12/16 16:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:46:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:46:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:46:21 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 16:46:26 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 16:46:31 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 16:46:36 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 16:46:36 App: TestKVRebInWithTimers got un-deployed
2021/12/16 16:46:36 No eventing-con process running
2021/12/16 16:46:36 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/16 16:46:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:46:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:46:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:46:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:46:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (270.27s)
=== RUN   TestKVRebOutWithTimers
2021/12/16 16:46:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 16:46:55 Removing node(s): [] from the cluster
2021/12/16 16:47:00 Rebalance progress: 100
2021/12/16 16:47:05 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 16:47:05 App: TestKVRebOutWithTimers got un-deployed
2021/12/16 16:47:05 No eventing-con process running
2021/12/16 16:47:05 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/16 16:47:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:47:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:47:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:47:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:47:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:47:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:47:24 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:47:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 16:47:30 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 16:47:35 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 16:47:40 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 16:47:45 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 16:47:45 App: TestKVRebOutWithTimers got deployed
2021/12/16 16:47:45 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:47:45 Starting bucket ops to source bucket
2021/12/16 16:47:50 Rebalance progress: 12.5
2021/12/16 16:47:55 Rebalance progress: 12.5
2021/12/16 16:48:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:48:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:48:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:48:00 Waiting for src bucket item count to get to: 50000 curr count: 19459
2021/12/16 16:48:05 Waiting for src bucket item count to get to: 50000 curr count: 19459
2021/12/16 16:48:10 Waiting for src bucket item count to get to: 50000 curr count: 29410
2021/12/16 16:48:15 Waiting for src bucket item count to get to: 50000 curr count: 29410
2021/12/16 16:48:20 Waiting for src bucket item count to get to: 50000 curr count: 41786
2021/12/16 16:48:25 Waiting for src bucket item count to get to: 50000 curr count: 41786
2021/12/16 16:48:30 Waiting for src bucket item count to get to: 50000 curr count: 41786
2021/12/16 16:48:40 Waiting for dst bucket item count to get to: 50000 curr count: 40721
2021/12/16 16:48:45 Waiting for dst bucket item count to get to: 50000 curr count: 40721
2021/12/16 16:48:50 Waiting for dst bucket item count to get to: 50000 curr count: 48930
2021/12/16 16:48:55 Waiting for dst bucket item count to get to: 50000 curr count: 48930
2021/12/16 16:48:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:48:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:48:55 Starting bucket ops to source bucket
2021/12/16 16:50:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:50:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:50:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:50:20 Waiting for dst bucket item count to get to: 0 curr count: 47487
2021/12/16 16:50:25 Waiting for dst bucket item count to get to: 0 curr count: 47487
2021/12/16 16:50:30 Waiting for dst bucket item count to get to: 0 curr count: 31824
2021/12/16 16:50:35 Waiting for dst bucket item count to get to: 0 curr count: 31824
2021/12/16 16:50:40 Waiting for dst bucket item count to get to: 0 curr count: 9597
2021/12/16 16:50:45 Waiting for dst bucket item count to get to: 0 curr count: 9597
2021/12/16 16:50:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:50:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:50:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:50:50 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 16:50:55 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 16:51:00 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 16:51:05 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 16:51:05 App: TestKVRebOutWithTimers got un-deployed
2021/12/16 16:51:05 No eventing-con process running
2021/12/16 16:51:05 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/16 16:51:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:51:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:51:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:51:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:51:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.04s)
=== RUN   TestKVSwapRebWithTimers
2021/12/16 16:51:29 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:51:29 App: TestKVSwapRebWithTimers got un-deployed
2021/12/16 16:51:29 No eventing-con process running
2021/12/16 16:51:29 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/16 16:51:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:51:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:51:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:51:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:51:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:51:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:51:47 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:51:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 16:51:54 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 16:51:59 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 16:52:04 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 16:52:09 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 16:52:09 App: TestKVSwapRebWithTimers got deployed
2021/12/16 16:52:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 16:52:09 Starting bucket ops to source bucket
2021/12/16 16:52:09 Removing node(s): [] from the cluster
2021/12/16 16:52:14 Rebalance progress: 12.5
2021/12/16 16:52:19 Rebalance progress: 12.5
2021/12/16 16:52:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:52:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 16:52:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:52:30 Rebalance progress: 12.5
2021/12/16 16:52:35 Rebalance progress: 12.5
2021/12/16 16:52:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:52:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:52:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:52:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 16:52:45 Rebalance progress: 12.5
2021/12/16 16:52:50 Rebalance progress: 12.5
2021/12/16 16:52:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:52:55 Waiting for src bucket item count to get to: 50000 curr count: 47439
2021/12/16 16:53:00 Waiting for src bucket item count to get to: 50000 curr count: 47439
2021/12/16 16:53:10 Waiting for dst bucket item count to get to: 50000 curr count: 39841
2021/12/16 16:53:15 Waiting for dst bucket item count to get to: 50000 curr count: 39841
2021/12/16 16:53:20 Waiting for dst bucket item count to get to: 50000 curr count: 48281
2021/12/16 16:53:25 Waiting for dst bucket item count to get to: 50000 curr count: 48281
2021/12/16 16:53:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:53:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:53:25 Starting bucket ops to source bucket
2021/12/16 16:54:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:54:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:54:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:54:50 Waiting for dst bucket item count to get to: 0 curr count: 48210
2021/12/16 16:54:55 Waiting for dst bucket item count to get to: 0 curr count: 48210
2021/12/16 16:55:00 Waiting for dst bucket item count to get to: 0 curr count: 36197
2021/12/16 16:55:05 Waiting for dst bucket item count to get to: 0 curr count: 36197
2021/12/16 16:55:10 Waiting for dst bucket item count to get to: 0 curr count: 16448
2021/12/16 16:55:15 Waiting for dst bucket item count to get to: 0 curr count: 16448
2021/12/16 16:55:20 Waiting for dst bucket item count to get to: 0 curr count: 4455
2021/12/16 16:55:25 Waiting for dst bucket item count to get to: 0 curr count: 4455
2021/12/16 16:55:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:55:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 16:55:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:55:30 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:55:35 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:55:40 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:55:45 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:55:50 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:55:55 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 16:55:55 App: TestKVSwapRebWithTimers got un-deployed
2021/12/16 16:55:56 No eventing-con process running
2021/12/16 16:55:56 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/16 16:55:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:56:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:56:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:56:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:56:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:56:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (290.17s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/16 16:56:19 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:56:19 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/16 16:56:19 No eventing-con process running
2021/12/16 16:56:19 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/16 16:56:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:56:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:56:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:56:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:56:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:56:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:56:38 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:56:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 16:56:44 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 16:56:49 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 16:56:54 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 16:56:59 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 16:56:59 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/16 16:56:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 16:56:59 Starting bucket ops to source bucket
2021/12/16 16:56:59 Removing node(s): [] from the cluster
2021/12/16 16:57:04 Rebalance progress: 12.5
2021/12/16 16:57:09 Rebalance progress: 12.5
2021/12/16 16:57:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:57:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:57:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:57:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 16:57:20 Rebalance progress: 12.5
2021/12/16 16:57:25 Rebalance progress: 12.5
2021/12/16 16:57:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:57:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:57:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 16:57:30 Waiting for src bucket item count to get to: 50000 curr count: 26132
2021/12/16 16:57:35 Waiting for src bucket item count to get to: 50000 curr count: 37003
2021/12/16 16:57:40 Waiting for src bucket item count to get to: 50000 curr count: 37003
2021/12/16 16:57:45 Waiting for src bucket item count to get to: 50000 curr count: 47793
2021/12/16 16:57:50 Waiting for src bucket item count to get to: 50000 curr count: 47793
2021/12/16 16:58:00 Waiting for dst bucket item count to get to: 50000 curr count: 37938
2021/12/16 16:58:05 Waiting for dst bucket item count to get to: 50000 curr count: 37938
2021/12/16 16:58:10 Waiting for dst bucket item count to get to: 50000 curr count: 45255
2021/12/16 16:58:15 Waiting for dst bucket item count to get to: 50000 curr count: 45255
2021/12/16 16:58:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 16:58:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 16:58:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 16:58:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:58:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:58:30 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:58:35 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:58:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:58:45 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 16:58:45 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/16 16:58:45 No eventing-con process running
2021/12/16 16:58:45 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/16 16:58:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:58:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:58:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:58:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:58:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:59:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:59:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 16:59:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.51s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/16 16:59:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 16:59:14 Removing node(s): [] from the cluster
2021/12/16 16:59:19 Rebalance progress: 100
2021/12/16 16:59:24 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 16:59:24 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/16 16:59:24 No eventing-con process running
2021/12/16 16:59:24 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/16 16:59:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:59:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:59:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 16:59:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:59:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:59:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 16:59:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 16:59:42 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 16:59:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 16:59:49 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 16:59:54 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 16:59:59 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 17:00:04 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 17:00:04 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/16 17:00:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:00:04 Starting bucket ops to source bucket
2021/12/16 17:00:09 Rebalance progress: 12.5
2021/12/16 17:00:14 Rebalance progress: 12.5
2021/12/16 17:00:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:00:19 Waiting for src bucket item count to get to: 50000 curr count: 6750
2021/12/16 17:00:24 Waiting for src bucket item count to get to: 50000 curr count: 20359
2021/12/16 17:00:29 Waiting for src bucket item count to get to: 50000 curr count: 20359
2021/12/16 17:00:34 Waiting for src bucket item count to get to: 50000 curr count: 34406
2021/12/16 17:00:39 Waiting for src bucket item count to get to: 50000 curr count: 34406
2021/12/16 17:00:44 Waiting for src bucket item count to get to: 50000 curr count: 44828
2021/12/16 17:00:49 Waiting for src bucket item count to get to: 50000 curr count: 44828
2021/12/16 17:00:54 Waiting for src bucket item count to get to: 50000 curr count: 44828
2021/12/16 17:01:04 Waiting for dst bucket item count to get to: 50000 curr count: 28985
2021/12/16 17:01:09 Waiting for dst bucket item count to get to: 50000 curr count: 28985
2021/12/16 17:01:14 Waiting for dst bucket item count to get to: 50000 curr count: 38461
2021/12/16 17:01:19 Waiting for dst bucket item count to get to: 50000 curr count: 38461
2021/12/16 17:01:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:01:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:01:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:01:24 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 17:01:29 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 17:01:34 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 17:01:39 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 17:01:44 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 17:01:44 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/16 17:01:44 No eventing-con process running
2021/12/16 17:01:44 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/16 17:01:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:01:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:01:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:01:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:02:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:02:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:02:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.26s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/16 17:02:18 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:02:18 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/16 17:02:18 No eventing-con process running
2021/12/16 17:02:18 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/16 17:02:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:02:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:02:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:02:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:02:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:02:37 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:02:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 17:02:43 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 17:02:48 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 17:02:53 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 17:02:58 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 17:02:58 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/16 17:02:58 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 17:02:58 Starting bucket ops to source bucket
2021/12/16 17:02:58 Removing node(s): [] from the cluster
2021/12/16 17:03:03 Rebalance progress: 12.5
2021/12/16 17:03:08 Rebalance progress: 12.5
2021/12/16 17:03:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:03:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:03:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:03:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 17:03:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:03:18 Rebalance progress: 12.5
2021/12/16 17:03:23 Rebalance progress: 12.5
2021/12/16 17:03:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:03:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:03:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:03:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:03:33 Rebalance progress: 12.5
2021/12/16 17:03:38 Rebalance progress: 12.5
2021/12/16 17:03:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:03:43 Waiting for src bucket item count to get to: 50000 curr count: 48904
2021/12/16 17:03:48 Waiting for src bucket item count to get to: 50000 curr count: 48904
2021/12/16 17:03:59 Waiting for dst bucket item count to get to: 50000 curr count: 35599
2021/12/16 17:04:04 Waiting for dst bucket item count to get to: 50000 curr count: 35599
2021/12/16 17:04:09 Waiting for dst bucket item count to get to: 50000 curr count: 41656
2021/12/16 17:04:14 Waiting for dst bucket item count to get to: 50000 curr count: 41656
2021/12/16 17:04:14 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:04:14 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:04:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:04:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:04:24 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:04:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:04:34 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:04:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:04:44 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 17:04:44 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/16 17:04:44 No eventing-con process running
2021/12/16 17:04:44 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/16 17:04:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:04:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:04:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:04:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:04:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:05:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:05:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:05:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.70s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/16 17:05:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:05:18 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/16 17:05:18 No eventing-con process running
2021/12/16 17:05:18 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/16 17:05:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:05:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:05:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:05:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:05:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:05:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:05:36 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:05:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 17:05:42 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 17:05:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 17:05:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 17:05:57 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 17:05:57 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/16 17:05:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:05:57 Starting bucket ops to source bucket
2021/12/16 17:05:58 Removing node(s): [] from the cluster
2021/12/16 17:06:03 Rebalance progress: 12.5
2021/12/16 17:06:08 Rebalance progress: 12.5
2021/12/16 17:06:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:06:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:06:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:06:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:06:18 Rebalance progress: 12.5
2021/12/16 17:06:23 Rebalance progress: 12.5
2021/12/16 17:06:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:06:28 Waiting for src bucket item count to get to: 50000 curr count: 27215
2021/12/16 17:06:33 Waiting for src bucket item count to get to: 50000 curr count: 38187
2021/12/16 17:06:38 Waiting for src bucket item count to get to: 50000 curr count: 38187
2021/12/16 17:06:43 Waiting for src bucket item count to get to: 50000 curr count: 49169
2021/12/16 17:06:48 Waiting for src bucket item count to get to: 50000 curr count: 49169
2021/12/16 17:06:58 Waiting for dst bucket item count to get to: 50000 curr count: 38456
2021/12/16 17:07:03 Waiting for dst bucket item count to get to: 50000 curr count: 38456
2021/12/16 17:07:08 Waiting for dst bucket item count to get to: 50000 curr count: 45789
2021/12/16 17:07:13 Waiting for dst bucket item count to get to: 50000 curr count: 45789
2021/12/16 17:07:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:07:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:07:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:07:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:07:23 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:07:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:07:33 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:07:38 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:07:43 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 17:07:43 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/16 17:07:43 No eventing-con process running
2021/12/16 17:07:43 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/16 17:07:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:07:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:07:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:07:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:08:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:08:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:08:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.32s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/16 17:08:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:08:12 Removing node(s): [] from the cluster
2021/12/16 17:08:17 Rebalance progress: 100
2021/12/16 17:08:22 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:08:22 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/16 17:08:22 No eventing-con process running
2021/12/16 17:08:22 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/16 17:08:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:08:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:08:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:08:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:08:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:08:41 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:08:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 17:08:47 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 17:08:52 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 17:08:57 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 17:09:02 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 17:09:02 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/16 17:09:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:09:02 Starting bucket ops to source bucket
2021/12/16 17:09:07 Rebalance progress: 12.5
2021/12/16 17:09:12 Rebalance progress: 12.5
2021/12/16 17:09:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:09:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:09:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:09:17 Waiting for src bucket item count to get to: 50000 curr count: 9279
2021/12/16 17:09:22 Waiting for src bucket item count to get to: 50000 curr count: 23089
2021/12/16 17:09:27 Waiting for src bucket item count to get to: 50000 curr count: 23089
2021/12/16 17:09:32 Waiting for src bucket item count to get to: 50000 curr count: 36516
2021/12/16 17:09:37 Waiting for src bucket item count to get to: 50000 curr count: 36516
2021/12/16 17:09:42 Waiting for src bucket item count to get to: 50000 curr count: 36516
2021/12/16 17:09:47 Waiting for src bucket item count to get to: 50000 curr count: 45163
2021/12/16 17:09:52 Waiting for src bucket item count to get to: 50000 curr count: 45163
2021/12/16 17:10:02 Waiting for dst bucket item count to get to: 50000 curr count: 29929
2021/12/16 17:10:07 Waiting for dst bucket item count to get to: 50000 curr count: 29929
2021/12/16 17:10:12 Waiting for dst bucket item count to get to: 50000 curr count: 43238
2021/12/16 17:10:17 Waiting for dst bucket item count to get to: 50000 curr count: 43238
2021/12/16 17:10:17 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:10:17 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:10:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:10:22 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:10:27 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:10:33 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:10:38 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:10:43 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:10:48 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 17:10:48 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/16 17:10:48 No eventing-con process running
2021/12/16 17:10:48 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/16 17:10:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:10:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:10:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:10:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:11:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:11:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.43s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/16 17:11:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:11:16 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/16 17:11:16 No eventing-con process running
2021/12/16 17:11:16 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/16 17:11:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:11:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:11:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:11:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:11:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:11:35 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:11:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 17:11:41 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 17:11:46 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 17:11:51 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 17:11:56 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 17:11:56 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/16 17:11:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:11:56 Starting bucket ops to source bucket
2021/12/16 17:11:56 Removing node(s): [] from the cluster
2021/12/16 17:12:01 Rebalance progress: 12.5
2021/12/16 17:12:06 Rebalance progress: 12.5
2021/12/16 17:12:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:12:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:12:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:12:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:12:17 Rebalance progress: 12.5
2021/12/16 17:12:22 Rebalance progress: 12.5
2021/12/16 17:12:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:12:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:12:32 Rebalance progress: 12.5
2021/12/16 17:12:37 Rebalance progress: 12.5
2021/12/16 17:12:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:12:42 Waiting for src bucket item count to get to: 50000 curr count: 49971
2021/12/16 17:12:47 Waiting for src bucket item count to get to: 50000 curr count: 49971
2021/12/16 17:12:57 Waiting for dst bucket item count to get to: 50000 curr count: 35489
2021/12/16 17:13:02 Waiting for dst bucket item count to get to: 50000 curr count: 35489
2021/12/16 17:13:07 Waiting for dst bucket item count to get to: 50000 curr count: 45841
2021/12/16 17:13:12 Waiting for dst bucket item count to get to: 50000 curr count: 45841
2021/12/16 17:13:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:13:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:13:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:13:17 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:13:22 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:13:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:13:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:13:38 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:13:43 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 17:13:43 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/16 17:13:43 No eventing-con process running
2021/12/16 17:13:43 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/16 17:13:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:13:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:13:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:13:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:13:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:14:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:14:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:14:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.97s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/16 17:14:16 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:14:16 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/16 17:14:16 No eventing-con process running
2021/12/16 17:14:16 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/16 17:14:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:14:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:14:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:14:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:14:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:14:35 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:14:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 17:14:41 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 17:14:46 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 17:14:51 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 17:14:56 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 17:14:56 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/16 17:14:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 17:14:56 Starting bucket ops to source bucket
2021/12/16 17:14:57 Removing node(s): [] from the cluster
2021/12/16 17:15:02 Rebalance progress: 12.5
2021/12/16 17:15:07 Rebalance progress: 12.5
2021/12/16 17:15:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:15:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:15:17 Rebalance progress: 12.5
2021/12/16 17:15:22 Rebalance progress: 12.5
2021/12/16 17:15:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:15:27 Waiting for src bucket item count to get to: 50000 curr count: 27390
2021/12/16 17:15:32 Waiting for src bucket item count to get to: 50000 curr count: 38920
2021/12/16 17:15:37 Waiting for src bucket item count to get to: 50000 curr count: 38920
2021/12/16 17:15:47 Waiting for dst bucket item count to get to: 50000 curr count: 37321
2021/12/16 17:15:52 Waiting for dst bucket item count to get to: 50000 curr count: 37321
2021/12/16 17:15:57 Waiting for dst bucket item count to get to: 50000 curr count: 47441
2021/12/16 17:16:02 Waiting for dst bucket item count to get to: 50000 curr count: 47441
2021/12/16 17:16:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:16:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:16:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:16:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:16:12 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:16:17 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:16:22 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:16:27 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:16:32 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 17:16:32 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/16 17:16:32 No eventing-con process running
2021/12/16 17:16:32 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/16 17:16:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:16:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:16:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:16:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:16:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:16:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:17:01 Waiting for dst bucket item count to get to: 0 curr count: 33984
2021/12/16 17:17:06 Waiting for dst bucket item count to get to: 0 curr count: 33984
2021/12/16 17:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.40s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/16 17:17:11 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 17:17:11 Removing node(s): [] from the cluster
2021/12/16 17:17:16 Rebalance progress: 100
2021/12/16 17:17:21 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:17:21 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/16 17:17:21 No eventing-con process running
2021/12/16 17:17:21 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/16 17:17:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:17:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:17:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:17:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:17:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:17:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:17:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:17:40 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:17:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 17:17:46 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 17:17:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 17:17:56 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 17:18:01 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 17:18:01 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/16 17:18:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:18:01 Starting bucket ops to source bucket
2021/12/16 17:18:06 Rebalance progress: 12.5
2021/12/16 17:18:11 Rebalance progress: 12.5
2021/12/16 17:18:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:18:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:18:16 Waiting for src bucket item count to get to: 50000 curr count: 11075
2021/12/16 17:18:21 Waiting for src bucket item count to get to: 50000 curr count: 24454
2021/12/16 17:18:26 Waiting for src bucket item count to get to: 50000 curr count: 24454
2021/12/16 17:18:31 Waiting for src bucket item count to get to: 50000 curr count: 35420
2021/12/16 17:18:36 Waiting for src bucket item count to get to: 50000 curr count: 35420
2021/12/16 17:18:41 Waiting for src bucket item count to get to: 50000 curr count: 46317
2021/12/16 17:18:46 Waiting for src bucket item count to get to: 50000 curr count: 46317
2021/12/16 17:18:51 Waiting for src bucket item count to get to: 50000 curr count: 46317
2021/12/16 17:19:01 Waiting for dst bucket item count to get to: 50000 curr count: 35957
2021/12/16 17:19:06 Waiting for dst bucket item count to get to: 50000 curr count: 35957
2021/12/16 17:19:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:19:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:19:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:19:11 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:19:16 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:19:21 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:19:26 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:19:32 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:19:37 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 17:19:37 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/16 17:19:37 No eventing-con process running
2021/12/16 17:19:37 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/16 17:19:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:19:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:19:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:19:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:19:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:19:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:20:00 Waiting for dst bucket item count to get to: 0 curr count: 15558
2021/12/16 17:20:05 Waiting for dst bucket item count to get to: 0 curr count: 15558
2021/12/16 17:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.55s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/16 17:20:15 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 17:20:15 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/16 17:20:15 No eventing-con process running
2021/12/16 17:20:15 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/16 17:20:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:20:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:20:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:20:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:20:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:20:34 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:20:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 17:20:40 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 17:20:45 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 17:20:50 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 17:20:55 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 17:20:55 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/16 17:20:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 17:20:55 Starting bucket ops to source bucket
2021/12/16 17:20:55 Removing node(s): [] from the cluster
2021/12/16 17:21:00 Rebalance progress: 12.5
2021/12/16 17:21:05 Rebalance progress: 12.5
2021/12/16 17:21:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:21:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:21:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:21:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 17:21:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:21:16 Rebalance progress: 12.5
2021/12/16 17:21:21 Rebalance progress: 12.5
2021/12/16 17:21:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:21:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:21:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:21:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:21:31 Rebalance progress: 12.5
2021/12/16 17:21:36 Rebalance progress: 12.5
2021/12/16 17:21:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:21:46 Waiting for dst bucket item count to get to: 50000 curr count: 36303
2021/12/16 17:21:51 Waiting for dst bucket item count to get to: 50000 curr count: 36303
2021/12/16 17:21:56 Waiting for dst bucket item count to get to: 50000 curr count: 43840
2021/12/16 17:22:01 Waiting for dst bucket item count to get to: 50000 curr count: 43840
2021/12/16 17:22:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:22:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:22:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:22:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 17:22:11 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 17:22:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 17:22:22 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 17:22:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 17:22:27 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/16 17:22:27 No eventing-con process running
2021/12/16 17:22:27 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/16 17:22:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:22:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:22:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:22:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:22:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:22:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:22:50 Waiting for dst bucket item count to get to: 0 curr count: 14617
2021/12/16 17:22:55 Waiting for dst bucket item count to get to: 0 curr count: 14617
2021/12/16 17:22:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.98s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/16 17:23:05 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:23:05 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/16 17:23:05 No eventing-con process running
2021/12/16 17:23:05 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/16 17:23:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:23:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:23:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:23:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:23:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:23:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:23:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:23:24 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:23:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 17:23:30 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 17:23:35 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 17:23:40 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 17:23:45 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 17:23:45 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/16 17:23:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:23:45 Starting bucket ops to source bucket
2021/12/16 17:23:46 Removing node(s): [] from the cluster
2021/12/16 17:23:51 Rebalance progress: 12.5
2021/12/16 17:23:56 Rebalance progress: 12.5
2021/12/16 17:24:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:24:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:24:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:24:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:24:06 Rebalance progress: 12.5
2021/12/16 17:24:11 Rebalance progress: 12.5
2021/12/16 17:24:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:24:16 Waiting for src bucket item count to get to: 50000 curr count: 29455
2021/12/16 17:24:21 Waiting for src bucket item count to get to: 50000 curr count: 40994
2021/12/16 17:24:26 Waiting for src bucket item count to get to: 50000 curr count: 40994
2021/12/16 17:24:36 Waiting for dst bucket item count to get to: 50000 curr count: 33785
2021/12/16 17:24:41 Waiting for dst bucket item count to get to: 50000 curr count: 33785
2021/12/16 17:24:46 Waiting for dst bucket item count to get to: 50000 curr count: 42203
2021/12/16 17:24:51 Waiting for dst bucket item count to get to: 50000 curr count: 42203
2021/12/16 17:24:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:24:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:24:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:24:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:25:01 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:25:06 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:25:11 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:25:16 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:25:21 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 17:25:21 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/16 17:25:21 No eventing-con process running
2021/12/16 17:25:21 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/16 17:25:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:25:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:25:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:25:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:25:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:25:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:25:50 Waiting for dst bucket item count to get to: 0 curr count: 3665
2021/12/16 17:25:55 Waiting for dst bucket item count to get to: 0 curr count: 3665
2021/12/16 17:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.79s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/16 17:26:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:26:00 Removing node(s): [] from the cluster
2021/12/16 17:26:05 Rebalance progress: 100
2021/12/16 17:26:10 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 17:26:10 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/16 17:26:10 No eventing-con process running
2021/12/16 17:26:10 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/16 17:26:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:26:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:26:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:26:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:26:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:26:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:26:29 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:26:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 17:26:35 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 17:26:40 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 17:26:45 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 17:26:50 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 17:26:50 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/16 17:26:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:26:50 Starting bucket ops to source bucket
2021/12/16 17:26:55 Rebalance progress: 12.5
2021/12/16 17:27:00 Rebalance progress: 12.5
2021/12/16 17:27:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:27:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:27:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:27:05 Waiting for src bucket item count to get to: 50000 curr count: 12684
2021/12/16 17:27:10 Waiting for src bucket item count to get to: 50000 curr count: 24449
2021/12/16 17:27:15 Waiting for src bucket item count to get to: 50000 curr count: 24449
2021/12/16 17:27:20 Waiting for src bucket item count to get to: 50000 curr count: 34283
2021/12/16 17:27:25 Waiting for src bucket item count to get to: 50000 curr count: 34283
2021/12/16 17:27:30 Waiting for src bucket item count to get to: 50000 curr count: 34283
2021/12/16 17:27:35 Waiting for src bucket item count to get to: 50000 curr count: 46109
2021/12/16 17:27:40 Waiting for src bucket item count to get to: 50000 curr count: 46109
2021/12/16 17:27:50 Waiting for dst bucket item count to get to: 50000 curr count: 44196
2021/12/16 17:27:55 Waiting for dst bucket item count to get to: 50000 curr count: 44196
2021/12/16 17:27:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:27:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:27:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:28:00 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 17:28:05 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 17:28:10 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 17:28:15 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 17:28:20 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 17:28:20 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/16 17:28:20 No eventing-con process running
2021/12/16 17:28:21 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/16 17:28:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:28:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:28:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:28:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:28:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:28:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:28:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:28:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (174.21s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/16 17:28:59 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:28:59 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/16 17:28:59 No eventing-con process running
2021/12/16 17:28:59 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/16 17:29:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:29:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:29:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:29:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:29:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:29:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:29:18 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:29:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 17:29:24 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 17:29:29 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 17:29:34 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 17:29:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 17:29:39 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/16 17:29:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:29:39 Starting bucket ops to source bucket
2021/12/16 17:29:40 Removing node(s): [] from the cluster
2021/12/16 17:29:45 Rebalance progress: 12.5
2021/12/16 17:29:50 Rebalance progress: 12.5
2021/12/16 17:29:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:29:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:29:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:29:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:29:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:30:00 Rebalance progress: 12.5
2021/12/16 17:30:05 Rebalance progress: 12.5
2021/12/16 17:30:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:30:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:30:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:30:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:30:15 Rebalance progress: 12.5
2021/12/16 17:30:20 Rebalance progress: 12.5
2021/12/16 17:30:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:30:25 Waiting for src bucket item count to get to: 50000 curr count: 45869
2021/12/16 17:30:30 Waiting for src bucket item count to get to: 50000 curr count: 45869
2021/12/16 17:30:40 Waiting for dst bucket item count to get to: 50000 curr count: 44037
2021/12/16 17:30:45 Waiting for dst bucket item count to get to: 50000 curr count: 44037
2021/12/16 17:30:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 17:30:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 17:30:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:30:50 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:30:55 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:31:00 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:31:05 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:31:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:31:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 17:31:16 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/16 17:31:16 No eventing-con process running
2021/12/16 17:31:16 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/16 17:31:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:31:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:31:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:31:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:31:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:31:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:31:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 17:31:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.89s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/16 17:31:49 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:31:49 App: bucket_op_on_update got un-deployed
2021/12/16 17:31:49 No eventing-con process running
2021/12/16 17:31:49 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:31:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:31:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:31:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:31:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:32:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:32:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:32:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:32:08 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:32:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:32:14 Starting bucket ops to source bucket
2021/12/16 17:32:19 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:32:24 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:32:29 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:32:29 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:32:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:32:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:32:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:32:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:32:34 Rebalance progress: 12.5
2021/12/16 17:32:39 Rebalance progress: 12.5
2021/12/16 17:32:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:32:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:32:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:32:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:32:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:32:50 Rebalance progress: 12.5
2021/12/16 17:32:55 Rebalance progress: 12.5
2021/12/16 17:33:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:33:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 17:33:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:33:05 Rebalance progress: 12.5
2021/12/16 17:33:10 Rebalance progress: 12.5
2021/12/16 17:33:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:33:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:33:20 Rebalance progress: 12.5
2021/12/16 17:33:25 Rebalance progress: 12.5
2021/12/16 17:33:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:33:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:33:35 Rebalance progress: 12.5
2021/12/16 17:33:40 Rebalance progress: 12.5
2021/12/16 17:33:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:33:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:33:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 17:33:50 Rebalance progress: 12.5
2021/12/16 17:33:55 Rebalance progress: 12.5
2021/12/16 17:34:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:34:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:34:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:34:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:34:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:34:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:34:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:34:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:34:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:34:31 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:34:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:34:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:34:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:34:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10635
2021/12/16 17:34:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10635
2021/12/16 17:34:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:34:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:34:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:34:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (195.30s)
=== RUN   TestKVForceFailover
2021/12/16 17:35:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:35:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:35:09 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:35:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:35:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:35:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:35:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:35:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:35:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:35:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:35:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:35:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:35:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:35:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:35:29 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:35:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:35:30 Starting bucket ops to source bucket
2021/12/16 17:35:35 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:35:40 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:35:45 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:35:50 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:35:50 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:35:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:35:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:35:50 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/16 17:36:10 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/16 17:36:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 17:36:15 Rebalance progress: 12.5
2021/12/16 17:36:20 Rebalance progress: 12.5
2021/12/16 17:36:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:36:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 17:36:31 Rebalance progress: 12.5
2021/12/16 17:36:36 Rebalance progress: 12.5
2021/12/16 17:36:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:36:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:36:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:36:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:36:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:36:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:36:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:37:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:37:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:37:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:37:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:37:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:37:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:37:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:37:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7052
2021/12/16 17:37:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:37:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:37:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:37:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (150.11s)
=== RUN   TestKVHardFailoverRecovery
2021/12/16 17:37:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:37:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:37:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:37:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:37:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:37:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:37:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:37:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:37:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:37:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:37:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:37:53 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:37:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:37:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:38:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:38:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:38:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:38:14 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:38:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:38:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:38:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:38:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:38:20 Rebalance progress: 12.5
2021/12/16 17:38:25 Rebalance progress: 12.5
2021/12/16 17:38:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:38:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:38:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:38:30 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/16 17:39:30 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/16 17:39:35 Rebalance progress: 12.5
2021/12/16 17:39:40 Rebalance progress: 12.5
2021/12/16 17:39:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:39:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:39:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:39:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:39:50 Rebalance progress: 12.5
2021/12/16 17:39:55 Rebalance progress: 12.5
2021/12/16 17:40:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:40:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:40:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:40:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:40:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:40:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:40:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:40:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:40:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:40:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:40:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:40:30 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:40:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:40:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:40:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:40:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12046
2021/12/16 17:40:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12046
2021/12/16 17:40:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:40:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:40:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:40:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12046
2021/12/16 17:40:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (214.49s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/16 17:41:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:41:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:41:09 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:41:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:41:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:41:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:41:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:41:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:41:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:41:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:41:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:41:28 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:41:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:41:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:41:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:41:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:41:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:41:49 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:41:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:41:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:41:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:41:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:41:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 17:41:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:41:55 Rebalance progress: 12.5
2021/12/16 17:42:00 Rebalance progress: 12.5
2021/12/16 17:42:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:42:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:42:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:42:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 17:42:10 Rebalance progress: 12.5
2021/12/16 17:42:15 Rebalance progress: 12.5
2021/12/16 17:42:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:42:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:42:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:42:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:42:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:42:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:42:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:42:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:42:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:42:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:42:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:42:50 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:42:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:42:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:42:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:43:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4619
2021/12/16 17:43:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4619
2021/12/16 17:43:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:43:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:43:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:43:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4619
2021/12/16 17:43:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.90s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/16 17:43:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:43:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:43:29 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:43:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:43:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:43:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:43:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:43:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:43:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:43:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:43:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:43:42 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:43:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:43:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:43:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:43:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:44:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:44:04 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:44:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:44:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:44:09 Rebalance progress: 12.5
2021/12/16 17:44:14 Rebalance progress: 12.5
2021/12/16 17:44:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:44:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:44:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:44:25 Rebalance progress: 12.5
2021/12/16 17:44:30 Rebalance progress: 12.5
2021/12/16 17:44:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:44:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:44:40 Rebalance progress: 12.5
2021/12/16 17:44:45 Rebalance progress: 12.5
2021/12/16 17:44:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:44:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:44:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:44:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:20 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:45:20 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:45:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:45:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:45:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:45:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6144
2021/12/16 17:45:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6144
2021/12/16 17:45:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:45:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:45:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:45:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6144
2021/12/16 17:45:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:45:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:45:54 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:45:54 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:45:54 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:45:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:46:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.92s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/16 17:46:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:46:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:46:18 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:46:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:46:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:46:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:46:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:46:36 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 17:46:36 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:46:36 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/16 17:46:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:46:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:50 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:46:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:46:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:46:51 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:46:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 17:46:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:47:02 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:47:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:47:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:47:12 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/16 17:47:17 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 17:47:22 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 17:47:27 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 17:47:27 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:47:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:47:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:47:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:47:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:47:33 Rebalance progress: 12.5
2021/12/16 17:47:38 Rebalance progress: 12.5
2021/12/16 17:47:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:47:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:47:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:47:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:47:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:47:48 Rebalance progress: 12.5
2021/12/16 17:47:53 Rebalance progress: 12.5
2021/12/16 17:47:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:47:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 17:47:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:48:04 Rebalance progress: 12.5
2021/12/16 17:48:09 Rebalance progress: 12.5
2021/12/16 17:48:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:48:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:48:19 Rebalance progress: 12.5
2021/12/16 17:48:24 Rebalance progress: 12.5
2021/12/16 17:48:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:48:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:48:34 Rebalance progress: 12.5
2021/12/16 17:48:39 Rebalance progress: 12.5
2021/12/16 17:48:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:48:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 17:48:49 Rebalance progress: 12.5
2021/12/16 17:48:54 Rebalance progress: 12.5
2021/12/16 17:48:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:48:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:48:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:49:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:49:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:49:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:49:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:49:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:49:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:49:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12192
2021/12/16 17:49:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:49:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:49:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:50:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1371
2021/12/16 17:50:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1371
2021/12/16 17:50:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:50:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:50:10 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:50:15 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:50:20 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:50:25 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:50:30 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:50:35 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:50:35 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 17:50:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:50:35 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/16 17:50:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:50:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:50:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:50:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:50:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:50:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (281.26s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/16 17:50:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:50:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:50:59 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:50:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:51:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:51:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:51:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:51:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:51:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:51:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:51:18 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:51:18 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 17:51:18 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:51:18 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/16 17:51:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:51:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:51:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:51:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:51:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:51:31 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:51:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:51:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:51:32 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:51:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 17:51:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:51:44 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:51:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:51:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:51:54 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/16 17:51:59 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 17:52:04 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 17:52:09 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 17:52:09 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:52:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:52:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:52:14 Rebalance progress: 12.5
2021/12/16 17:52:19 Rebalance progress: 12.5
2021/12/16 17:52:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:52:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:52:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:52:30 Rebalance progress: 12.5
2021/12/16 17:52:35 Rebalance progress: 12.5
2021/12/16 17:52:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:52:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:52:45 Rebalance progress: 12.5
2021/12/16 17:52:50 Rebalance progress: 12.5
2021/12/16 17:52:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:52:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:53:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:53:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:53:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:53:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:53:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:53:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:53:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:53:25 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:53:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:53:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:53:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:53:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7616
2021/12/16 17:53:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:53:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:53:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:53:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7616
2021/12/16 17:53:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7616
2021/12/16 17:53:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:53:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:54:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:54:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:54:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:54:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:54:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:54:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 17:54:26 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 17:54:26 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:54:26 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/16 17:54:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:54:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:54:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:54:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:54:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:54:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (231.00s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 17:54:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:54:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:54:50 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:54:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:54:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:54:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:54:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:54:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:55:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:55:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:55:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:55:09 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:55:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:55:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:55:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:55:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:55:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:55:30 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:55:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:55:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:55:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:55:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:55:50 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:55:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:55:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:56:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:56:06 Rebalance progress: 12.5
2021/12/16 17:56:11 Rebalance progress: 12.5
2021/12/16 17:56:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:56:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:56:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:56:21 Rebalance progress: 12.5
2021/12/16 17:56:26 Rebalance progress: 12.5
2021/12/16 17:56:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:56:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:56:36 Rebalance progress: 12.5
2021/12/16 17:56:41 Rebalance progress: 12.5
2021/12/16 17:56:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:56:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:56:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:56:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:57:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:57:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:57:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:57:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:57:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:57:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:57:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 17:57:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:57:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:57:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9149
2021/12/16 17:57:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:57:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:57:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:57:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4511
2021/12/16 17:57:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4511
2021/12/16 17:57:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.34s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/16 17:57:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:57:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 17:57:55 eventingSetup.go:46: No eventing-con process running
2021/12/16 17:57:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 17:57:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:58:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 17:58:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:58:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 17:58:14 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 17:58:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 17:58:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 17:58:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:58:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:58:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 17:58:35 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:58:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 17:58:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:58:41 Rebalance progress: 12.5
2021/12/16 17:58:46 Rebalance progress: 12.5
2021/12/16 17:58:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:58:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:58:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:59:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 17:59:06 Rebalance progress: 12.5
2021/12/16 17:59:11 Rebalance progress: 12.5
2021/12/16 17:59:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:59:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 17:59:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 17:59:21 Rebalance progress: 12.5
2021/12/16 17:59:26 Rebalance progress: 12.5
2021/12/16 17:59:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:59:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 17:59:36 Rebalance progress: 12.5
2021/12/16 17:59:41 Rebalance progress: 12.5
2021/12/16 17:59:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 17:59:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 17:59:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 17:59:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 17:59:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:00:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:00:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:00:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:00:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:00:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 18:00:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 18:00:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 18:00:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:00:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 18:00:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8594
2021/12/16 18:00:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:00:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:00:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 18:00:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1510
2021/12/16 18:00:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1510
2021/12/16 18:00:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.70s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/16 18:00:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 18:00:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 18:00:56 Rebalance progress: 100
2021/12/16 18:00:56 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 18:01:26 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 18:01:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 18:01:32 Removing node(s): [] from the cluster
2021/12/16 18:01:32 Starting bucket ops to source bucket
2021/12/16 18:01:37 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/16 18:01:42 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 18:01:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 18:01:47 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:01:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:01:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 18:01:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 18:01:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:01:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:02:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:02:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:02:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:02:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 18:02:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 18:02:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 18:02:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 18:02:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:02:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 18:02:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1503
2021/12/16 18:02:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 18:02:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 18:02:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 18:02:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1271
2021/12/16 18:02:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1271
2021/12/16 18:02:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.74s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12133.706s

Rebalance tests took 3 hours, 22 minutes and 16 seconds.

Testing Succeeded


Version: versions-16.12.2021-10.17.cfg
Build Log: make-16.12.2021-10.17.log
Server Log: logs-16.12.2021-10.17.tar.gz

Finished