Building

Started building at 2021/12/17 10:09:43
Calculating base
Updating mirror
Basing run on 7.1.0-1934 b4ad217fa
Updating tree for run 17.12.2021-10.09
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f0d7a1e, changes since last good build: none
kv_engine is at deb741c, changes since last good build: 
 deb741c0d ep_testsuite: Avoid unnecessary exception throwing
 77f539672 MB-49156: Use auto* instead of const CookieIFace* in tests
 3b08256d9 Add variadic template parameter to StoredDocKeyT
ns_server is at 8596843, changes since last good build: 
 859684340 MB-50057: Add CBC ciphers to low_security_ciphers list
 2c3701fc3 MB-49795 Better message when multi-node kv autofailover couldn't complete
couchstore is at 70053f3, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at a657fc6, changes since last good build: none
Switching eventing to unstable
eventing is at d9c8767, changes since last good build: 
 d9c8767f MB-49570: Add backlog stats on ui
 9061db3b MB-49570: Start dcp stream for a single collection id
 67ef79f4 MB-49720: Store failover log in checkpoint blob
Switching eventing-ee to unstable
eventing-ee is at 3f826e0, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable a431b08a] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 98c1e4b1] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/18
Total 4 (delta 3), reused 4 (delta 3)
[unstable 93a8da9e] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable cb644d23] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/17 10:26:53
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/17 10:28:36 Started functional
2021/12/17 10:28:47 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/17 10:28:53 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/17 10:28:53 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 10:28:57 Removing node(s): [] from the cluster
2021/12/17 10:29:02 Rebalance progress: 0
2021/12/17 10:29:07 Rebalance progress: 0.3074363425925926
2021/12/17 10:29:12 Rebalance progress: 4.973234953703703
2021/12/17 10:29:17 Rebalance progress: 9.25925925925926
2021/12/17 10:29:22 Rebalance progress: 12.98466435185185
2021/12/17 10:29:27 Rebalance progress: 17.59620949074074
2021/12/17 10:29:32 Rebalance progress: 20.86950231481481
2021/12/17 10:29:37 Rebalance progress: 25.46296296296296
2021/12/17 10:29:42 Rebalance progress: 100
2021/12/17 10:29:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:30:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:30:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:30:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:30:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:31:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:31:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:31:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:31:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:32:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:32:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:32:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:32:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:33:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:33:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:33:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/17 10:33:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/17 10:34:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:34:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:34:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:34:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:35:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:35:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:35:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:35:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:36:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:36:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:36:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:36:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:37:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:37:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:37:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:37:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/17 10:38:03 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:38:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:38:09 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 10:38:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 10:38:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 10:38:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/17 10:38:24 App: TestAdvancedGetOps got deployed
2021/12/17 10:38:24 Starting bucket ops to source bucket
2021/12/17 10:38:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:38:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:38:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 10:38:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:38:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:38:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:38:39 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 10:38:45 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 10:38:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 10:38:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/17 10:38:55 App: TestAdvancedGetOps got un-deployed
2021/12/17 10:38:55 No eventing-con process running
2021/12/17 10:38:55 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/17 10:38:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:39:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:39:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:39:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:39:18 Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/17 10:39:23 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/17 10:39:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (320.18s)
=== RUN   TestAdvancedInsertOps
2021/12/17 10:39:23 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:39:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:39:29 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:39:34 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:39:39 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:39:44 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:39:44 App: TestAdvancedInsertOps got deployed
2021/12/17 10:39:44 Starting bucket ops to source bucket
2021/12/17 10:39:50 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 10:39:55 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/17 10:39:55 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/17 10:39:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:39:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:39:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:40:00 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:40:05 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:40:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:40:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:40:15 App: TestAdvancedInsertOps got un-deployed
2021/12/17 10:40:15 No eventing-con process running
2021/12/17 10:40:15 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/17 10:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:40:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:40:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:40:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:40:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:40:44 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 10:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:40:44 Testing insert operation on source bucket
2021/12/17 10:40:44 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:40:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:40:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:40:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:41:00 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:41:05 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/17 10:41:05 App: TestAdvancedInsertOps got deployed
2021/12/17 10:41:05 Starting bucket ops to source bucket
2021/12/17 10:41:11 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/17 10:41:11 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/17 10:41:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:41:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:41:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:41:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:41:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:41:26 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:41:31 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/17 10:41:31 App: TestAdvancedInsertOps got un-deployed
2021/12/17 10:41:31 No eventing-con process running
2021/12/17 10:41:31 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/17 10:41:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:41:43 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/17 10:41:48 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:41:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:41:50 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:41:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.57s)
=== RUN   TestAdvancedUpsertOps
2021/12/17 10:41:55 Starting bucket ops to source bucket
2021/12/17 10:41:55 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:41:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:42:01 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:42:06 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:42:11 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:42:16 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:42:17 App: TestAdvancedUpsertOps got deployed
2021/12/17 10:42:17 Starting bucket ops to source bucket
2021/12/17 10:42:23 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:42:28 Waiting for dst bucket item count to get to: 0 curr count: 170
2021/12/17 10:42:33 Waiting for dst bucket item count to get to: 0 curr count: 170
2021/12/17 10:42:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:42:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:42:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:42:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:42:38 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:42:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:42:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:42:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:42:53 App: TestAdvancedUpsertOps got un-deployed
2021/12/17 10:42:53 No eventing-con process running
2021/12/17 10:42:53 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/17 10:42:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:43:04 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:43:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:43:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:43:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:43:11 Testing upsert operation on source bucket
2021/12/17 10:43:11 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:43:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:43:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:43:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:43:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:43:32 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/17 10:43:32 App: TestAdvancedUpsertOps got deployed
2021/12/17 10:43:32 Starting bucket ops to source bucket
2021/12/17 10:43:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:43:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:43:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:43:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:43:39 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:43:44 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:43:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:43:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/17 10:43:54 App: TestAdvancedUpsertOps got un-deployed
2021/12/17 10:43:54 No eventing-con process running
2021/12/17 10:43:54 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/17 10:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:44:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:44:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:44:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.93s)
=== RUN   TestAdvancedReplaceOps
2021/12/17 10:44:08 Starting bucket ops to source bucket
2021/12/17 10:44:09 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:44:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:44:15 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:44:20 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:44:25 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:44:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:44:30 App: TestAdvancedReplaceOps got deployed
2021/12/17 10:44:30 Starting bucket ops to source bucket
2021/12/17 10:44:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:44:41 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 10:44:46 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 10:44:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:44:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:44:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:44:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:44:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:44:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:45:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:45:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:45:06 App: TestAdvancedReplaceOps got un-deployed
2021/12/17 10:45:06 No eventing-con process running
2021/12/17 10:45:06 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/17 10:45:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:45:18 Waiting for dst bucket item count to get to: 0 curr count: 543
2021/12/17 10:45:23 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/17 10:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:45:29 Testing upsert operation on source bucket
2021/12/17 10:45:29 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:45:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:45:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:45:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:45:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:45:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/17 10:45:51 App: TestAdvancedReplaceOps got deployed
2021/12/17 10:45:51 Starting bucket ops to source bucket
2021/12/17 10:45:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:45:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:45:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:45:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:45:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:46:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:46:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:46:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/17 10:46:12 App: TestAdvancedReplaceOps got un-deployed
2021/12/17 10:46:12 No eventing-con process running
2021/12/17 10:46:12 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/17 10:46:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:46:24 Waiting for dst bucket item count to get to: 0 curr count: 170
2021/12/17 10:46:29 Waiting for dst bucket item count to get to: 0 curr count: 94
2021/12/17 10:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:46:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (147.99s)
=== RUN   TestAdvancedDeleteOps
2021/12/17 10:46:36 Starting bucket ops to source bucket
2021/12/17 10:46:36 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:46:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:46:42 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:46:47 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:46:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:46:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:46:57 App: TestAdvancedDeleteOps got deployed
2021/12/17 10:46:57 Starting bucket ops to source bucket
2021/12/17 10:47:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:47:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:47:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:47:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:47:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:47:07 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:47:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:47:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:47:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:47:22 App: TestAdvancedDeleteOps got un-deployed
2021/12/17 10:47:22 No eventing-con process running
2021/12/17 10:47:22 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/17 10:47:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:47:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:47:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 10:47:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:47:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:47:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:47:46 Testing delete operation on source bucket
2021/12/17 10:47:46 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:47:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/17 10:47:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:47:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:48:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:48:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/17 10:48:07 App: TestAdvancedDeleteOps got deployed
2021/12/17 10:48:07 Starting bucket ops to source bucket
2021/12/17 10:48:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:48:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:48:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:48:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:48:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:48:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:48:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:48:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/17 10:48:27 App: TestAdvancedDeleteOps got un-deployed
2021/12/17 10:48:27 No eventing-con process running
2021/12/17 10:48:27 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/17 10:48:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:48:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:48:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:48:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.62s)
=== RUN   TestEnoentAdvancedGet
2021/12/17 10:48:41 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:48:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/17 10:48:47 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:48:52 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:48:57 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:49:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:49:02 App: TestEnoentAdvancedGet got deployed
2021/12/17 10:49:02 Starting bucket ops to source bucket
2021/12/17 10:49:08 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:49:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:49:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 10:49:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:49:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:49:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:49:18 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:49:23 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:49:28 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:49:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:49:33 App: TestEnoentAdvancedGet got un-deployed
2021/12/17 10:49:33 No eventing-con process running
2021/12/17 10:49:33 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/17 10:49:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:49:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:49:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:49:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:49:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:49:56 Testing get enoent operation on source bucket
2021/12/17 10:49:56 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:49:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/17 10:50:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:50:07 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:50:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:50:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/17 10:50:17 App: TestEnoentAdvancedGet got deployed
2021/12/17 10:50:17 Starting bucket ops to source bucket
2021/12/17 10:50:23 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/17 10:50:28 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 10:50:33 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 10:50:33 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/17 10:50:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:50:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:50:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:50:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:50:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:50:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:50:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/17 10:50:53 App: TestEnoentAdvancedGet got un-deployed
2021/12/17 10:50:53 No eventing-con process running
2021/12/17 10:50:53 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/17 10:50:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:51:05 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/17 10:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:51:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (150.15s)
=== RUN   TestEnoentAdvancedReplace
2021/12/17 10:51:11 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:51:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 10:51:17 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:51:22 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:51:27 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:51:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:51:33 App: TestEnoentAdvancedReplace got deployed
2021/12/17 10:51:33 Starting bucket ops to source bucket
2021/12/17 10:51:38 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:51:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:51:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 10:51:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:51:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:51:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:51:48 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:51:53 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:51:58 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:52:03 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:52:03 App: TestEnoentAdvancedReplace got un-deployed
2021/12/17 10:52:03 No eventing-con process running
2021/12/17 10:52:03 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/17 10:52:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:52:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:52:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:52:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:52:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:52:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:52:26 Testing get enoent operation on source bucket
2021/12/17 10:52:26 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:52:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 10:52:33 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:52:38 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:52:43 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:52:48 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/17 10:52:48 App: TestEnoentAdvancedReplace got deployed
2021/12/17 10:52:48 Starting bucket ops to source bucket
2021/12/17 10:52:53 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/17 10:52:58 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 10:53:03 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 10:53:03 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/17 10:53:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:53:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:53:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:53:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:53:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:53:18 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:53:23 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/17 10:53:23 App: TestEnoentAdvancedReplace got un-deployed
2021/12/17 10:53:23 No eventing-con process running
2021/12/17 10:53:23 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/17 10:53:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:53:35 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/17 10:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:53:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:53:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.07s)
=== RUN   TestEnoentAdvancedDelete
2021/12/17 10:53:41 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:53:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 10:53:48 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:53:53 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:53:58 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:54:03 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:54:03 App: TestEnoentAdvancedDelete got deployed
2021/12/17 10:54:03 Starting bucket ops to source bucket
2021/12/17 10:54:08 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:54:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 10:54:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 10:54:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:54:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:54:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:54:18 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:54:23 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:54:28 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:54:33 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:54:33 App: TestEnoentAdvancedDelete got un-deployed
2021/12/17 10:54:33 No eventing-con process running
2021/12/17 10:54:33 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/17 10:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:54:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:54:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:54:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:54:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:54:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:54:56 Testing get enoent operation on source bucket
2021/12/17 10:54:56 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:54:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 10:55:03 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:55:08 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:55:13 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:55:18 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/17 10:55:18 App: TestEnoentAdvancedDelete got deployed
2021/12/17 10:55:18 Starting bucket ops to source bucket
2021/12/17 10:55:23 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/17 10:55:28 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 10:55:33 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/17 10:55:33 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/17 10:55:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:55:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:55:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:55:38 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:55:43 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:55:48 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:55:53 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/17 10:55:53 App: TestEnoentAdvancedDelete got un-deployed
2021/12/17 10:55:53 No eventing-con process running
2021/12/17 10:55:53 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/17 10:55:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:56:05 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/17 10:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:56:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.08s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/17 10:56:12 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:56:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/17 10:56:18 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:56:23 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:56:28 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:56:33 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:56:33 App: TestAdvancedInsertKeyExist got deployed
2021/12/17 10:56:33 Starting bucket ops to source bucket
2021/12/17 10:56:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:56:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:56:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:56:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:56:38 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:56:43 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:56:48 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:56:53 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:56:53 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/17 10:56:53 No eventing-con process running
2021/12/17 10:56:53 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/17 10:56:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:57:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:57:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:57:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:57:07 Pumping document in the destination bucket
2021/12/17 10:57:07 Starting bucket ops to source bucket
2021/12/17 10:57:07 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:57:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/17 10:57:13 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:57:18 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:57:23 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:57:28 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/17 10:57:28 App: TestAdvancedInsertKeyExist got deployed
2021/12/17 10:57:28 Starting bucket ops to source bucket
2021/12/17 10:57:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 10:57:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 10:57:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:57:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:57:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:57:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:57:44 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:57:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:57:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:57:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/17 10:57:59 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/17 10:57:59 No eventing-con process running
2021/12/17 10:57:59 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/17 10:58:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:58:11 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/17 10:58:16 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/17 10:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:58:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:58:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.76s)
=== RUN   TestExipryGet
2021/12/17 10:58:22 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:58:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/17 10:58:28 Waiting for app: TestExipryGet to get deployed
2021/12/17 10:58:33 Waiting for app: TestExipryGet to get deployed
2021/12/17 10:58:38 Waiting for app: TestExipryGet to get deployed
2021/12/17 10:58:44 Waiting for app: TestExipryGet to get deployed
2021/12/17 10:58:44 App: TestExipryGet got deployed
2021/12/17 10:58:44 Starting bucket ops to source bucket
2021/12/17 10:59:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:59:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 10:59:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 10:59:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 10:59:29 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 10:59:34 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 10:59:39 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 10:59:44 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 10:59:44 App: TestExipryGet got un-deployed
2021/12/17 10:59:44 No eventing-con process running
2021/12/17 10:59:44 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/17 10:59:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:59:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:59:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 10:59:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 10:59:58 Pumping document in the destination bucket for expiry get
2021/12/17 10:59:58 Starting bucket ops to source bucket
2021/12/17 10:59:58 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 10:59:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/17 11:00:04 Waiting for app: TestExipryGet to get deployed
2021/12/17 11:00:09 Waiting for app: TestExipryGet to get deployed
2021/12/17 11:00:14 Waiting for app: TestExipryGet to get deployed
2021/12/17 11:00:19 Waiting for app: TestExipryGet to get deployed
2021/12/17 11:00:19 App: TestExipryGet got deployed
2021/12/17 11:00:19 Starting bucket ops to source bucket
2021/12/17 11:01:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:01:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:01:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:01:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:01:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:01:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:01:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:01:20 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 11:01:25 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 11:01:30 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 11:01:35 Waiting for app: TestExipryGet to get un-deployed
2021/12/17 11:01:35 App: TestExipryGet got un-deployed
2021/12/17 11:01:35 No eventing-con process running
2021/12/17 11:01:35 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/17 11:01:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:01:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:01:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:01:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:01:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.82s)
=== RUN   TestExipryInsert
2021/12/17 11:01:53 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:01:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/17 11:01:59 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:02:04 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:02:09 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:02:14 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:02:14 App: TestExipryInsert got deployed
2021/12/17 11:02:14 Starting bucket ops to source bucket
2021/12/17 11:03:00 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/17 11:03:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:03:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:03:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:03:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:03:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:03:10 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:03:15 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:03:20 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:03:25 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:03:25 App: TestExipryInsert got un-deployed
2021/12/17 11:03:25 No eventing-con process running
2021/12/17 11:03:25 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/17 11:03:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:03:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:03:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:03:39 Testing on the destination bucket for expiry insert
2021/12/17 11:03:39 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:03:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/17 11:03:45 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:03:50 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:03:55 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:04:00 Waiting for app: TestExipryInsert to get deployed
2021/12/17 11:04:00 App: TestExipryInsert got deployed
2021/12/17 11:04:00 Starting bucket ops to source bucket
2021/12/17 11:04:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:04:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:04:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:04:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:04:50 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:04:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:04:55 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:05:00 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:05:05 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:05:10 Waiting for app: TestExipryInsert to get un-deployed
2021/12/17 11:05:10 App: TestExipryInsert got un-deployed
2021/12/17 11:05:10 No eventing-con process running
2021/12/17 11:05:10 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/17 11:05:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:05:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:05:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:05:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:05:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:05:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.82s)
=== RUN   TestExipryReplace
2021/12/17 11:05:34 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:05:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 11:05:40 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:05:45 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:05:50 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:05:55 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:05:55 App: TestExipryReplace got deployed
2021/12/17 11:05:55 Starting bucket ops to source bucket
2021/12/17 11:06:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:06:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:06:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:06:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:06:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:06:46 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:06:51 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:06:56 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:07:01 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:07:01 App: TestExipryReplace got un-deployed
2021/12/17 11:07:01 No eventing-con process running
2021/12/17 11:07:01 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/17 11:07:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:07:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:07:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:07:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:07:15 Testing on the destination bucket for expiry insert
2021/12/17 11:07:15 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:07:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 11:07:21 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:07:26 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:07:31 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:07:36 Waiting for app: TestExipryReplace to get deployed
2021/12/17 11:07:36 App: TestExipryReplace got deployed
2021/12/17 11:07:36 Starting bucket ops to source bucket
2021/12/17 11:08:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:08:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:08:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:08:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:08:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:08:26 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:08:31 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:08:36 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:08:41 Waiting for app: TestExipryReplace to get un-deployed
2021/12/17 11:08:41 App: TestExipryReplace got un-deployed
2021/12/17 11:08:41 No eventing-con process running
2021/12/17 11:08:41 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/17 11:08:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:08:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:08:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:08:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:09:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:09:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.65s)
=== RUN   TestExipryUpsert
2021/12/17 11:09:05 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:09:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/17 11:09:11 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:09:16 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:09:21 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:09:26 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:09:26 App: TestExipryUpsert got deployed
2021/12/17 11:09:26 Starting bucket ops to source bucket
2021/12/17 11:10:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:10:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:10:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:10:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:10:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:10:16 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:10:21 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:10:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:10:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:10:31 App: TestExipryUpsert got un-deployed
2021/12/17 11:10:31 No eventing-con process running
2021/12/17 11:10:31 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/17 11:10:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:10:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:10:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:10:45 Testing on the destination bucket for expiry insert
2021/12/17 11:10:45 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:10:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/17 11:10:51 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:10:56 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:11:01 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:11:06 Waiting for app: TestExipryUpsert to get deployed
2021/12/17 11:11:06 App: TestExipryUpsert got deployed
2021/12/17 11:11:06 Starting bucket ops to source bucket
2021/12/17 11:11:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:11:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:11:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:11:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:11:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:11:57 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:12:02 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:12:07 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:12:12 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/17 11:12:12 App: TestExipryUpsert got un-deployed
2021/12/17 11:12:12 No eventing-con process running
2021/12/17 11:12:12 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/17 11:12:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:12:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:12:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:12:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:12:30 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.81s)
=== RUN   TestCasReplace
2021/12/17 11:12:35 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:12:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 11:12:42 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:12:47 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:12:52 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:12:57 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:12:57 App: TestCasReplace got deployed
2021/12/17 11:12:57 Starting bucket ops to source bucket
2021/12/17 11:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:12:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:12:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:12:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:13:02 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:13:07 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:13:12 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:13:17 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:13:17 App: TestCasReplace got un-deployed
2021/12/17 11:13:17 No eventing-con process running
2021/12/17 11:13:17 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/17 11:13:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:13:29 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/17 11:13:34 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/17 11:13:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:13:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:13:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:13:40 Testing on the destination bucket for upsert with cas
2021/12/17 11:13:40 Starting bucket ops to source bucket
2021/12/17 11:13:41 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:13:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/17 11:13:47 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:13:52 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:13:57 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:14:02 Waiting for app: TestCasReplace to get deployed
2021/12/17 11:14:02 App: TestCasReplace got deployed
2021/12/17 11:14:02 Starting bucket ops to source bucket
2021/12/17 11:14:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:14:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:14:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:14:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:14:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:14:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:14:17 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:14:22 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:14:27 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:14:32 Waiting for app: TestCasReplace to get un-deployed
2021/12/17 11:14:32 App: TestCasReplace got un-deployed
2021/12/17 11:14:32 No eventing-con process running
2021/12/17 11:14:32 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/17 11:14:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:14:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:14:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:14:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:14:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (140.29s)
=== RUN   TestCasDelete
2021/12/17 11:14:56 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:14:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 11:15:02 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:15:07 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:15:12 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:15:17 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:15:17 App: TestCasDelete got deployed
2021/12/17 11:15:17 Starting bucket ops to source bucket
2021/12/17 11:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:15:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:15:17 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:15:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:15:22 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:15:27 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:15:32 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:15:37 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:15:37 App: TestCasDelete got un-deployed
2021/12/17 11:15:37 No eventing-con process running
2021/12/17 11:15:37 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/17 11:15:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:15:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:15:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:15:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:15:51 Testing on the destination bucket for delete with cas
2021/12/17 11:15:51 Starting bucket ops to source bucket
2021/12/17 11:15:52 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:15:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 11:15:58 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:16:03 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:16:08 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:16:13 Waiting for app: TestCasDelete to get deployed
2021/12/17 11:16:13 App: TestCasDelete got deployed
2021/12/17 11:16:13 Starting bucket ops to source bucket
2021/12/17 11:16:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:16:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:16:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:16:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:16:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 11:16:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:16:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:16:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:16:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:16:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/17 11:16:43 App: TestCasDelete got un-deployed
2021/12/17 11:16:43 No eventing-con process running
2021/12/17 11:16:43 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/17 11:16:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:16:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:16:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (125.86s)
=== RUN   TestCountersIncrement
2021/12/17 11:17:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 11:17:07 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 11:17:08 Removing node(s): [] from the cluster
2021/12/17 11:17:13 Rebalance progress: 100
2021/12/17 11:17:13 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:17:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/17 11:17:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:17:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:17:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:17:34 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:17:34 App: TestCountersIncrement got deployed
2021/12/17 11:17:34 Starting bucket ops to source bucket
2021/12/17 11:17:40 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 11:17:45 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 11:17:45 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/17 11:17:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:17:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:17:50 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:17:55 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:18:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:18:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:18:05 App: TestCountersIncrement got un-deployed
2021/12/17 11:18:05 No eventing-con process running
2021/12/17 11:18:06 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/17 11:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:18:17 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/17 11:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:18:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:18:29 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 11:18:34 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 11:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:18:34 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:18:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/17 11:18:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:18:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:18:50 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:18:55 Waiting for app: TestCountersIncrement to get deployed
2021/12/17 11:18:55 App: TestCountersIncrement got deployed
2021/12/17 11:18:55 Starting bucket ops to source bucket
2021/12/17 11:19:02 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/17 11:19:02 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/17 11:19:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:19:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:19:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:19:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:19:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:19:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/17 11:19:22 App: TestCountersIncrement got un-deployed
2021/12/17 11:19:22 No eventing-con process running
2021/12/17 11:19:22 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/17 11:19:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:19:34 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/17 11:19:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/17 11:19:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:19:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:19:50 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 11:19:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 11:19:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:19:55 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/17 11:20:01 Rebalance progress: 100
--- PASS: TestCountersIncrement (179.01s)
=== RUN   TestCountersDecrement
2021/12/17 11:20:01 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 11:20:01 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 11:20:01 Removing node(s): [] from the cluster
2021/12/17 11:20:06 Rebalance progress: 100
2021/12/17 11:20:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:20:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/17 11:20:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:20:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:20:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:20:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:20:27 App: TestCountersDecrement got deployed
2021/12/17 11:20:27 Starting bucket ops to source bucket
2021/12/17 11:20:34 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/17 11:20:39 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/17 11:20:44 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/17 11:20:44 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/17 11:20:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:20:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:20:49 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:20:54 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:20:59 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:21:04 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:21:04 App: TestCountersDecrement got un-deployed
2021/12/17 11:21:04 No eventing-con process running
2021/12/17 11:21:04 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/17 11:21:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:21:16 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/17 11:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:21:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:21:27 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/17 11:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:21:27 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:21:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/17 11:21:34 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:21:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:21:44 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:21:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/17 11:21:49 App: TestCountersDecrement got deployed
2021/12/17 11:21:49 Starting bucket ops to source bucket
2021/12/17 11:21:55 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/17 11:22:00 Waiting for dst bucket item count to get to: 2048 curr count: 522
2021/12/17 11:22:05 Waiting for dst bucket item count to get to: 2048 curr count: 1546
2021/12/17 11:22:05 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/17 11:22:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:22:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:22:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:22:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:22:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:22:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/17 11:22:25 App: TestCountersDecrement got un-deployed
2021/12/17 11:22:25 No eventing-con process running
2021/12/17 11:22:25 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/17 11:22:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:22:37 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/17 11:22:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:22:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:22:49 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 11:22:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:22:49 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/17 11:22:54 Rebalance progress: 100
--- PASS: TestCountersDecrement (173.07s)
=== RUN   TestCacheGet
2021/12/17 11:22:54 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:22:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/17 11:23:00 Waiting for app: TestCacheGet to get deployed
2021/12/17 11:23:05 Waiting for app: TestCacheGet to get deployed
2021/12/17 11:23:10 Waiting for app: TestCacheGet to get deployed
2021/12/17 11:23:15 Waiting for app: TestCacheGet to get deployed
2021/12/17 11:23:15 App: TestCacheGet got deployed
2021/12/17 11:23:15 Starting bucket ops to source bucket
2021/12/17 11:23:20 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/17 11:23:25 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/17 11:23:30 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/17 11:23:35 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/17 11:23:40 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/17 11:23:40 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/17 11:23:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:23:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:23:45 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 11:23:50 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 11:23:55 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 11:24:00 Waiting for app: TestCacheGet to get un-deployed
2021/12/17 11:24:00 App: TestCacheGet got un-deployed
2021/12/17 11:24:00 No eventing-con process running
2021/12/17 11:24:00 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/17 11:24:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:24:12 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/17 11:24:17 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/17 11:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:24:29 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/17 11:24:34 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/17 11:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.25s)
=== RUN   TestCacheRYOW
2021/12/17 11:24:34 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:24:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/17 11:24:40 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 11:24:45 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 11:24:50 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 11:24:55 Waiting for app: TestCacheRYOW to get deployed
2021/12/17 11:24:55 App: TestCacheRYOW got deployed
2021/12/17 11:24:55 Starting bucket ops to source bucket
2021/12/17 11:25:00 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/17 11:25:00 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/17 11:25:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:25:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:25:05 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 11:25:11 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 11:25:16 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 11:25:21 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/17 11:25:21 App: TestCacheRYOW got un-deployed
2021/12/17 11:25:21 No eventing-con process running
2021/12/17 11:25:21 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/17 11:25:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:25:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:25:37 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 11:25:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:25:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:25:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:25:54 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 11:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.29s)
=== RUN   TestCacheExpiry
2021/12/17 11:25:54 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:25:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/17 11:26:00 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 11:26:05 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 11:26:10 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 11:26:15 Waiting for app: TestCacheExpiry to get deployed
2021/12/17 11:26:15 App: TestCacheExpiry got deployed
2021/12/17 11:26:15 Starting bucket ops to source bucket
2021/12/17 11:26:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:26:26 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 11:26:31 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 11:26:36 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 11:26:41 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 11:26:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:26:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:26:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:26:46 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 11:26:51 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 11:26:56 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 11:27:01 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/17 11:27:01 App: TestCacheExpiry got un-deployed
2021/12/17 11:27:01 No eventing-con process running
2021/12/17 11:27:01 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/17 11:27:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:27:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:27:18 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 11:27:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:27:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:27:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:27:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:27:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.27s)
=== RUN   TestCacheExpiryCustom
2021/12/17 11:27:34 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:27:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/17 11:27:41 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 11:27:46 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 11:27:51 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 11:27:56 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/17 11:27:56 App: TestCacheExpiryCustom got deployed
2021/12/17 11:27:56 Starting bucket ops to source bucket
2021/12/17 11:28:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:28:06 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 11:28:11 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/17 11:28:16 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 11:28:21 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/17 11:28:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:28:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:28:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:28:26 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 11:28:31 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 11:28:36 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 11:28:41 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/17 11:28:41 App: TestCacheExpiryCustom got un-deployed
2021/12/17 11:28:41 No eventing-con process running
2021/12/17 11:28:41 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/17 11:28:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:28:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:28:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/17 11:28:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:29:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:29:10 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:29:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.26s)
=== RUN   TestCacheOverflowCustom
2021/12/17 11:29:15 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:29:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/17 11:29:21 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 11:29:26 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 11:29:31 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 11:29:36 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/17 11:29:36 App: TestCacheOverflowCustom got deployed
2021/12/17 11:29:36 Starting bucket ops to source bucket
2021/12/17 11:29:41 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/17 11:29:46 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/17 11:29:51 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/17 11:29:56 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/17 11:30:01 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/17 11:30:01 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/17 11:30:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:30:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:30:06 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 11:30:11 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 11:30:16 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 11:30:21 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/17 11:30:21 App: TestCacheOverflowCustom got un-deployed
2021/12/17 11:30:21 No eventing-con process running
2021/12/17 11:30:21 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/17 11:30:23 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:30:33 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/17 11:30:38 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/17 11:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:30:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:30:50 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/17 11:30:55 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/17 11:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.24s)
=== RUN   TestStrictMode
2021/12/17 11:30:55 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:30:56 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/17 11:30:56 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:30:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/17 11:31:02 Waiting for app: TestDataTypes to get deployed
2021/12/17 11:31:07 Waiting for app: TestDataTypes to get deployed
2021/12/17 11:31:12 Waiting for app: TestDataTypes to get deployed
2021/12/17 11:31:17 Waiting for app: TestDataTypes to get deployed
2021/12/17 11:31:17 App: TestDataTypes got deployed
2021/12/17 11:31:17 Starting bucket ops to source bucket
2021/12/17 11:31:22 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/17 11:31:27 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/17 11:31:32 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/17 11:31:32 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/17 11:31:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:31:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:31:37 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 11:31:42 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 11:31:47 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 11:31:52 Waiting for app: TestDataTypes to get un-deployed
2021/12/17 11:31:52 App: TestDataTypes got un-deployed
2021/12/17 11:31:52 No eventing-con process running
2021/12/17 11:31:53 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/17 11:31:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:32:04 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 11:32:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:32:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:32:16 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/17 11:32:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.12s)
=== RUN   TestEnoentGet
2021/12/17 11:32:16 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:32:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/17 11:32:22 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:32:27 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:32:32 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:32:37 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:32:37 App: TestEnoentGet got deployed
2021/12/17 11:32:37 Starting bucket ops to source bucket
2021/12/17 11:32:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:32:47 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:32:52 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:32:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:32:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:32:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:32:58 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:33:03 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:33:08 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:33:13 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:33:13 App: TestEnoentGet got un-deployed
2021/12/17 11:33:13 No eventing-con process running
2021/12/17 11:33:13 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/17 11:33:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:33:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:33:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:33:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:33:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:33:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:33:36 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:33:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/17 11:33:42 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:33:47 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:33:52 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:33:57 Waiting for app: TestEnoentGet to get deployed
2021/12/17 11:33:57 App: TestEnoentGet got deployed
2021/12/17 11:33:57 Starting bucket ops to source bucket
2021/12/17 11:34:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:34:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:34:13 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:34:13 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:34:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:34:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:34:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:34:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:34:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:34:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/17 11:34:33 App: TestEnoentGet got un-deployed
2021/12/17 11:34:33 No eventing-con process running
2021/12/17 11:34:33 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/17 11:34:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:34:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:34:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:34:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:34:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.38s)
=== RUN   TestEnoentDelete
2021/12/17 11:34:56 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:34:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 11:35:03 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:35:08 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:35:13 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:35:18 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:35:18 App: TestEnoentDelete got deployed
2021/12/17 11:35:18 Starting bucket ops to source bucket
2021/12/17 11:35:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:35:28 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:35:33 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:35:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:35:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:35:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:35:38 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:35:43 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:35:48 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:35:53 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:35:53 App: TestEnoentDelete got un-deployed
2021/12/17 11:35:53 No eventing-con process running
2021/12/17 11:35:53 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/17 11:35:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:36:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:36:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:36:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:36:17 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:36:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 11:36:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:36:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:36:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:36:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/17 11:36:38 App: TestEnoentDelete got deployed
2021/12/17 11:36:38 Starting bucket ops to source bucket
2021/12/17 11:36:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:36:48 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:36:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/17 11:36:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:36:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:36:58 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:37:03 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:37:08 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:37:13 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/17 11:37:13 App: TestEnoentDelete got un-deployed
2021/12/17 11:37:13 No eventing-con process running
2021/12/17 11:37:13 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/17 11:37:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:37:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:37:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:37:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:37:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.31s)
=== RUN   TestError
2021/12/17 11:37:47 Waiting for app: TestError to get un-deployed
2021/12/17 11:37:47 App: TestError got un-deployed
2021/12/17 11:37:47 No eventing-con process running
2021/12/17 11:37:47 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/17 11:37:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:38:00 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:38:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/17 11:38:07 Waiting for app: TestError to get deployed
2021/12/17 11:38:12 Waiting for app: TestError to get deployed
2021/12/17 11:38:17 Waiting for app: TestError to get deployed
2021/12/17 11:38:22 Waiting for app: TestError to get deployed
2021/12/17 11:38:22 App: TestError got deployed
2021/12/17 11:38:22 Starting bucket ops to source bucket
2021/12/17 11:38:27 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/17 11:38:32 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/17 11:38:32 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/17 11:38:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:38:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:38:37 Waiting for app: TestError to get un-deployed
2021/12/17 11:38:42 Waiting for app: TestError to get un-deployed
2021/12/17 11:38:47 Waiting for app: TestError to get un-deployed
2021/12/17 11:38:52 Waiting for app: TestError to get un-deployed
2021/12/17 11:38:52 App: TestError got un-deployed
2021/12/17 11:38:52 No eventing-con process running
2021/12/17 11:38:52 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/17 11:38:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:39:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:39:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:39:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:39:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:39:21 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/17 11:39:26 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/17 11:39:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.81s)
=== RUN   TestCRLF
2021/12/17 11:39:36 Waiting for app: TestCRLF to get un-deployed
2021/12/17 11:39:36 App: TestCRLF got un-deployed
2021/12/17 11:39:36 No eventing-con process running
2021/12/17 11:39:36 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/17 11:39:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:39:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:39:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:39:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:39:49 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:39:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/17 11:39:55 Waiting for app: TestCRLF to get deployed
2021/12/17 11:40:00 Waiting for app: TestCRLF to get deployed
2021/12/17 11:40:05 Waiting for app: TestCRLF to get deployed
2021/12/17 11:40:10 Waiting for app: TestCRLF to get deployed
2021/12/17 11:40:10 App: TestCRLF got deployed
2021/12/17 11:40:10 Starting bucket ops to source bucket
2021/12/17 11:40:16 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:40:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 11:40:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 11:40:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:40:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:40:26 Waiting for app: TestCRLF to get un-deployed
2021/12/17 11:40:31 Waiting for app: TestCRLF to get un-deployed
2021/12/17 11:40:36 Waiting for app: TestCRLF to get un-deployed
2021/12/17 11:40:41 Waiting for app: TestCRLF to get un-deployed
2021/12/17 11:40:41 App: TestCRLF got un-deployed
2021/12/17 11:40:41 No eventing-con process running
2021/12/17 11:40:41 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/17 11:40:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:40:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 11:40:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 11:40:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:41:09 Waiting for dst bucket item count to get to: 0 curr count: 48
2021/12/17 11:41:14 Waiting for dst bucket item count to get to: 0 curr count: 48
2021/12/17 11:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.77s)
=== RUN   TestImportExport
2021/12/17 11:41:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 11:41:24 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/17 11:41:24 No eventing-con process running
2021/12/17 11:41:24 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/17 11:41:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:41:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:41:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:41:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:41:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 11:41:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/17 11:41:43 No eventing-con process running
2021/12/17 11:41:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/17 11:41:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:41:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:41:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:41:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:41:57 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:41:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 11:41:58 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:41:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 11:42:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 11:42:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 11:42:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 11:42:19 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/17 11:42:19 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/17 11:42:24 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/17 11:42:29 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/17 11:42:34 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/17 11:42:34 App: TestImportExport_bucket_op_on_update got deployed
2021/12/17 11:42:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:42:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 11:42:44 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 11:42:49 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 11:42:54 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 11:42:54 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/17 11:42:54 No eventing-con process running
2021/12/17 11:42:54 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/17 11:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:42:59 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 11:43:04 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 11:43:09 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 11:43:14 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 11:43:14 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/17 11:43:14 No eventing-con process running
2021/12/17 11:43:14 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/17 11:43:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/17 11:43:32 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/17 11:43:32 No eventing-con process running
2021/12/17 11:43:32 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/17 11:43:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:43:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:43:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:43:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/17 11:43:50 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/17 11:43:50 No eventing-con process running
2021/12/17 11:43:50 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/17 11:43:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:43:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:43:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:44:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.61s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/17 11:44:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:44:14 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 11:44:14 No eventing-con process running
2021/12/17 11:44:14 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/17 11:44:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:44:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:44:23 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:44:28 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:44:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:44:29 Starting bucket ops to source bucket
2021/12/17 11:44:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:44:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:44:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:44:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:44:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 11:44:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:45:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 11:45:02 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 11:45:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:45:02 Undeploying app: bucket_op_on_update
2021/12/17 11:45:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:45:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:45:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:45:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:45:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:45:22 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 11:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:45:22 No eventing-con process running
2021/12/17 11:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:45:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:46:06 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:46:07 Starting bucket ops to source bucket
2021/12/17 11:46:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:46:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:46:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:46:30 Waiting for dst bucket item count to get to: 5000 curr count: 1415
2021/12/17 11:46:35 Waiting for dst bucket item count to get to: 5000 curr count: 1415
2021/12/17 11:46:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:46:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 11:46:40 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 11:46:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:46:40 Undeploying app: bucket_op_on_update
2021/12/17 11:46:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:46:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:46:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:46:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:47:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:47:01 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 11:47:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:47:01 No eventing-con process running
2021/12/17 11:47:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:47:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:47:45 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:47:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:47:46 Starting bucket ops to source bucket
2021/12/17 11:47:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:47:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:48:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:48:09 Waiting for dst bucket item count to get to: 5000 curr count: 2232
2021/12/17 11:48:14 Waiting for dst bucket item count to get to: 5000 curr count: 2232
2021/12/17 11:48:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:48:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 11:48:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 11:48:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:48:19 Undeploying app: bucket_op_on_update
2021/12/17 11:48:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:48:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:48:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:48:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:48:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:48:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 11:48:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:48:39 No eventing-con process running
2021/12/17 11:48:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:48:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:49:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:49:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:49:24 Starting bucket ops to source bucket
2021/12/17 11:49:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:49:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:49:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:49:47 Waiting for dst bucket item count to get to: 5000 curr count: 1698
2021/12/17 11:49:52 Waiting for dst bucket item count to get to: 5000 curr count: 4198
2021/12/17 11:49:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:49:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 11:49:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 11:49:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:49:57 Undeploying app: bucket_op_on_update
2021/12/17 11:49:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:50:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:50:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:50:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:50:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:50:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 11:50:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:50:17 No eventing-con process running
2021/12/17 11:50:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:50:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:51:01 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:51:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:51:02 Starting bucket ops to source bucket
2021/12/17 11:51:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:51:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:51:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:51:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:51:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:51:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/17 11:51:30 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/17 11:51:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:51:30 Undeploying app: bucket_op_on_update
2021/12/17 11:51:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:51:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:51:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:51:45 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:51:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/17 11:51:51 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/17 11:51:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:51:51 No eventing-con process running
2021/12/17 11:51:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:52:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:52:35 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.68s)
=== RUN   TestOnUpdateN1QLOp
2021/12/17 11:52:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 11:52:45 App: TestOnUpdateN1QLOp got un-deployed
2021/12/17 11:52:45 No eventing-con process running
2021/12/17 11:52:45 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/17 11:52:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:52:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:52:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:52:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:52:58 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:52:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/17 11:52:59 Starting bucket ops to source bucket
2021/12/17 11:53:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:53:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:53:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:53:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:53:27 Waiting for dst bucket item count to get to: 5000 curr count: 2480
2021/12/17 11:53:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:53:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:53:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:53:33 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 11:53:38 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 11:53:43 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 11:53:48 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/17 11:53:48 App: TestOnUpdateN1QLOp got un-deployed
2021/12/17 11:53:48 No eventing-con process running
2021/12/17 11:53:48 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/17 11:53:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:53:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:54:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:54:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 11:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.57s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/17 11:54:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 11:54:26 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/17 11:54:26 No eventing-con process running
2021/12/17 11:54:26 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/17 11:54:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:54:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:54:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:54:40 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:54:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:54:41 Starting bucket ops to source bucket
2021/12/17 11:54:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:54:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:54:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:55:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:55:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:55:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:55:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:55:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 11:55:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 11:55:19 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 11:55:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/17 11:55:24 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/17 11:55:24 No eventing-con process running
2021/12/17 11:55:24 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/17 11:55:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:55:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 11:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:55:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:55:48 Waiting for dst bucket item count to get to: 0 curr count: 3129
2021/12/17 11:55:53 Waiting for dst bucket item count to get to: 0 curr count: 629
2021/12/17 11:55:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.59s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/17 11:56:03 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 11:56:03 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 11:56:03 No eventing-con process running
2021/12/17 11:56:03 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/17 11:56:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:56:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:56:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:56:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:56:16 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:56:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 11:56:18 Starting bucket ops to source bucket
2021/12/17 11:56:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:56:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:56:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:56:41 Waiting for dst bucket item count to get to: 5000 curr count: 196
2021/12/17 11:56:46 Waiting for dst bucket item count to get to: 5000 curr count: 196
2021/12/17 11:56:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:56:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:56:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:56:51 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 11:56:56 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 11:57:01 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 11:57:06 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 11:57:06 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 11:57:06 No eventing-con process running
2021/12/17 11:57:06 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/17 11:57:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:57:18 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:57:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:57:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:57:30 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:57:35 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:57:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.71s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/17 11:57:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 11:57:45 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/17 11:57:45 No eventing-con process running
2021/12/17 11:57:45 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/17 11:57:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:57:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:57:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:57:58 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:57:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/17 11:57:59 Starting bucket ops to source bucket
2021/12/17 11:58:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:58:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:58:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:58:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:58:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 11:58:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 11:58:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 11:58:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 11:58:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 11:58:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 11:58:42 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 11:58:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/17 11:58:48 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/17 11:58:48 No eventing-con process running
2021/12/17 11:58:48 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/17 11:58:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:58:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:59:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 11:59:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:59:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:59:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 11:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.64s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/17 11:59:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 11:59:26 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/17 11:59:26 No eventing-con process running
2021/12/17 11:59:26 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/17 11:59:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:59:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:59:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 11:59:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 11:59:40 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/17 11:59:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/17 11:59:41 Starting bucket ops to source bucket
2021/12/17 11:59:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:59:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 11:59:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:00:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:00:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:00:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:00:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:00:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 12:00:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 12:00:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 12:00:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/17 12:00:25 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/17 12:00:25 No eventing-con process running
2021/12/17 12:00:25 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/17 12:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:00:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:00:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:00:48 Waiting for dst bucket item count to get to: 0 curr count: 3984
2021/12/17 12:00:53 Waiting for dst bucket item count to get to: 0 curr count: 1484
2021/12/17 12:00:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.13s)
=== RUN   TestOnDeleteBucketOp
2021/12/17 12:01:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 12:01:03 App: TestOnDeleteBucketOp got un-deployed
2021/12/17 12:01:03 No eventing-con process running
2021/12/17 12:01:03 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/17 12:01:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:01:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:01:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:01:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:01:17 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:01:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 12:01:18 Starting bucket ops to source bucket
2021/12/17 12:01:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:01:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:01:39 Waiting for dst bucket item count to get to: 5000 curr count: 128
2021/12/17 12:01:44 Waiting for dst bucket item count to get to: 5000 curr count: 128
2021/12/17 12:01:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:01:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:01:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:01:49 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 12:01:54 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 12:01:59 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 12:02:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/17 12:02:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/17 12:02:04 No eventing-con process running
2021/12/17 12:02:04 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/17 12:02:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:02:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:02:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:02:23 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:02:28 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/17 12:02:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.98s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/17 12:02:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 12:02:38 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/17 12:02:38 No eventing-con process running
2021/12/17 12:02:38 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/17 12:02:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:02:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:02:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:02:52 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:02:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/17 12:02:53 Starting bucket ops to source bucket
2021/12/17 12:03:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:03:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:03:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:03:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:03:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:03:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:03:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 12:03:24 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 12:03:29 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 12:03:34 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/17 12:03:34 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/17 12:03:34 No eventing-con process running
2021/12/17 12:03:34 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/17 12:03:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:03:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:03:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:03:58 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/17 12:03:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.03s)
=== RUN   TestTimerBucketOp
2021/12/17 12:04:08 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 12:04:08 App: TestTimerBucketOp got un-deployed
2021/12/17 12:04:08 No eventing-con process running
2021/12/17 12:04:08 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/17 12:04:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:04:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:04:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:04:22 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:04:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 12:04:23 Starting bucket ops to source bucket
2021/12/17 12:04:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:04:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:04:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:04:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:04:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:04:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:05:01 Waiting for dst bucket item count to get to: 5000 curr count: 1514
2021/12/17 12:05:06 Waiting for dst bucket item count to get to: 5000 curr count: 3324
2021/12/17 12:05:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:05:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:05:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:05:11 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 12:05:16 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 12:05:21 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 12:05:26 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/17 12:05:26 App: TestTimerBucketOp got un-deployed
2021/12/17 12:05:26 No eventing-con process running
2021/12/17 12:05:26 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/17 12:05:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:05:38 Waiting for dst bucket item count to get to: 0 curr count: 3185
2021/12/17 12:05:43 Waiting for dst bucket item count to get to: 0 curr count: 685
2021/12/17 12:05:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:05:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.43s)
=== RUN   TestTimerInPastBucketOp
2021/12/17 12:06:05 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 12:06:05 App: TestTimerInPastBucketOp got un-deployed
2021/12/17 12:06:05 No eventing-con process running
2021/12/17 12:06:05 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/17 12:06:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:06:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:06:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:06:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:06:18 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:06:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 12:06:19 Starting bucket ops to source bucket
2021/12/17 12:06:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:06:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:06:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:06:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:06:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:06:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:06:57 Waiting for dst bucket item count to get to: 5000 curr count: 1487
2021/12/17 12:07:02 Waiting for dst bucket item count to get to: 5000 curr count: 3987
2021/12/17 12:07:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:07:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:07:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:07:08 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 12:07:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 12:07:18 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 12:07:23 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/17 12:07:23 App: TestTimerInPastBucketOp got un-deployed
2021/12/17 12:07:23 No eventing-con process running
2021/12/17 12:07:23 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/17 12:07:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:07:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:07:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:07:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:07:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.50s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/17 12:07:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:07:56 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 12:07:56 No eventing-con process running
2021/12/17 12:07:56 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/17 12:07:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:08:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:08:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:08:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:08:10 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:08:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 12:08:11 Starting bucket ops to source bucket
2021/12/17 12:08:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:08:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:08:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:08:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:08:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:08:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:08:49 Waiting for dst bucket item count to get to: 5000 curr count: 2068
2021/12/17 12:08:54 Waiting for dst bucket item count to get to: 5000 curr count: 2068
2021/12/17 12:08:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:08:59 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 12:08:59 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 12:08:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:08:59 Undeploying app: bucket_op_with_timer
2021/12/17 12:08:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:09:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:09:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:09:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:09:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:09:19 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 12:09:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:09:19 No eventing-con process running
2021/12/17 12:09:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:09:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:10:03 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:10:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 12:10:05 Starting bucket ops to source bucket
2021/12/17 12:10:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:10:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:10:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 12:10:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 12:10:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:10:48 Undeploying app: bucket_op_with_timer
2021/12/17 12:10:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:10:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:10:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:11:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:11:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:11:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 12:11:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:11:08 No eventing-con process running
2021/12/17 12:11:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:11:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:11:52 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:11:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 12:11:53 Starting bucket ops to source bucket
2021/12/17 12:12:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:12:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:12:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:12:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:12:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:12:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:12:31 Waiting for dst bucket item count to get to: 5000 curr count: 21
2021/12/17 12:12:36 Waiting for dst bucket item count to get to: 5000 curr count: 2521
2021/12/17 12:12:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:12:41 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 12:12:41 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 12:12:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:12:41 Undeploying app: bucket_op_with_timer
2021/12/17 12:12:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:12:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:12:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:12:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:13:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:13:01 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 12:13:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:13:01 No eventing-con process running
2021/12/17 12:13:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:13:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:13:45 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:13:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 12:13:46 Starting bucket ops to source bucket
2021/12/17 12:13:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:13:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:14:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:14:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:14:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:14:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:14:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:14:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:14:29 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 12:14:29 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 12:14:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:14:29 Undeploying app: bucket_op_with_timer
2021/12/17 12:14:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:14:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:14:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:14:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:14:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:14:50 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 12:14:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:14:50 No eventing-con process running
2021/12/17 12:14:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:14:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:15:34 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:15:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 12:15:35 Starting bucket ops to source bucket
2021/12/17 12:15:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:15:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:15:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:15:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:16:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:16:08 Waiting for dst bucket item count to get to: 5000 curr count: 1071
2021/12/17 12:16:13 Waiting for dst bucket item count to get to: 5000 curr count: 1071
2021/12/17 12:16:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:16:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/17 12:16:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/17 12:16:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:16:18 Undeploying app: bucket_op_with_timer
2021/12/17 12:16:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:16:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:16:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:16:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:16:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/17 12:16:38 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/17 12:16:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:16:38 No eventing-con process running
2021/12/17 12:16:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:16:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:17:22 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (575.83s)
=== RUN   TestMultipleHandlers
2021/12/17 12:17:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 12:17:32 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/17 12:17:32 No eventing-con process running
2021/12/17 12:17:32 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/17 12:17:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:17:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:17:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:17:51 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 12:17:51 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/17 12:17:51 No eventing-con process running
2021/12/17 12:17:51 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/17 12:17:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:17:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:17:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:18:04 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:18:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 12:18:05 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:18:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/17 12:18:12 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 12:18:17 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 12:18:22 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 12:18:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/17 12:18:27 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/17 12:18:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/17 12:18:37 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/17 12:18:42 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/17 12:18:42 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/17 12:18:42 Starting bucket ops to source bucket
2021/12/17 12:18:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:18:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:18:52 Starting bucket ops to source bucket
2021/12/17 12:19:06 Waiting for dst bucket item count to get to: 10000 curr count: 6647
2021/12/17 12:19:12 Waiting for dst bucket item count to get to: 10000 curr count: 6647
2021/12/17 12:19:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 12:19:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:19:12 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:19:12 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:19:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:19:17 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 12:19:22 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 12:19:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 12:19:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/17 12:19:32 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/17 12:19:32 No eventing-con process running
2021/12/17 12:19:32 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/17 12:19:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:19:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:19:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:19:51 Waiting for dst bucket item count to get to: 0 curr count: 8956
2021/12/17 12:19:56 Waiting for dst bucket item count to get to: 0 curr count: 8956
2021/12/17 12:19:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:19:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:20:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 12:20:06 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 12:20:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 12:20:16 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/17 12:20:16 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/17 12:20:16 No eventing-con process running
2021/12/17 12:20:16 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/17 12:20:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:20:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:20:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:20:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.94s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/17 12:20:40 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 12:20:40 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/17 12:20:40 No eventing-con process running
2021/12/17 12:20:40 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/17 12:20:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:20:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:20:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:20:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:20:54 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:20:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 12:20:55 Starting bucket ops to source bucket
2021/12/17 12:21:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:21:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:21:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:21:18 Waiting for dst bucket item count to get to: 5000 curr count: 787
2021/12/17 12:21:23 Waiting for dst bucket item count to get to: 5000 curr count: 3287
2021/12/17 12:21:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:21:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:21:28 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 12:21:28 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 12:21:28 Pausing app: bucket_op_on_update
2021/12/17 12:21:28 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:21:33 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 12:21:33 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 12:21:33 Resuming app: bucket_op_on_update
2021/12/17 12:21:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:21:34 Starting bucket ops to source bucket
2021/12/17 12:21:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:21:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:21:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:21:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 12:21:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:21:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 12:21:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 12:21:57 Pausing app: bucket_op_on_update
2021/12/17 12:21:57 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:22:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 12:22:02 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 12:22:02 Resuming app: bucket_op_on_update
2021/12/17 12:22:04 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:22:04 Starting bucket ops to source bucket
2021/12/17 12:22:11 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 12:22:16 Waiting for dst bucket item count to get to: 15000 curr count: 10005
2021/12/17 12:22:21 Waiting for dst bucket item count to get to: 15000 curr count: 10005
2021/12/17 12:22:21 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/17 12:22:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:22:26 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 12:22:26 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 12:22:26 Pausing app: bucket_op_on_update
2021/12/17 12:22:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:22:32 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 12:22:32 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 12:22:32 Resuming app: bucket_op_on_update
2021/12/17 12:22:33 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:22:33 Starting bucket ops to source bucket
2021/12/17 12:22:41 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:22:46 Waiting for dst bucket item count to get to: 20000 curr count: 15012
2021/12/17 12:22:51 Waiting for dst bucket item count to get to: 20000 curr count: 15012
2021/12/17 12:22:56 Waiting for dst bucket item count to get to: 20000 curr count: 15012
2021/12/17 12:22:56 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/17 12:22:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:23:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 12:23:01 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 12:23:01 Pausing app: bucket_op_on_update
2021/12/17 12:23:01 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:23:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 12:23:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 12:23:06 Resuming app: bucket_op_on_update
2021/12/17 12:23:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:23:07 Starting bucket ops to source bucket
2021/12/17 12:23:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:23:20 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:23:25 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:23:25 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/17 12:23:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:23:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/17 12:23:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/17 12:23:30 Pausing app: bucket_op_on_update
2021/12/17 12:23:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:23:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/17 12:23:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/17 12:23:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:23:40 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 12:23:45 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 12:23:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 12:23:55 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/17 12:23:55 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/17 12:23:55 No eventing-con process running
2021/12/17 12:23:55 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/17 12:23:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:24:07 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 12:24:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:24:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:24:19 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 12:24:24 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/17 12:24:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.97s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/17 12:24:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 12:24:34 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/17 12:24:34 No eventing-con process running
2021/12/17 12:24:34 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/17 12:24:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:24:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:24:43 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:24:48 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:24:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 12:24:49 Starting bucket ops to source bucket
2021/12/17 12:24:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:25:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:25:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:25:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:25:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:25:22 Waiting for dst bucket item count to get to: 5000 curr count: 233
2021/12/17 12:25:27 Waiting for dst bucket item count to get to: 5000 curr count: 2733
2021/12/17 12:25:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:25:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:25:32 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 12:25:32 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 12:25:32 Pausing app: bucket_op_with_timer
2021/12/17 12:25:32 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:25:37 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 12:25:37 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 12:25:37 Resuming app: bucket_op_with_timer
2021/12/17 12:25:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:25:38 Starting bucket ops to source bucket
2021/12/17 12:25:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:25:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:25:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:26:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:26:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:26:11 Waiting for dst bucket item count to get to: 10000 curr count: 5230
2021/12/17 12:26:16 Waiting for dst bucket item count to get to: 10000 curr count: 7730
2021/12/17 12:26:16 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 12:26:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:26:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 12:26:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 12:26:21 Pausing app: bucket_op_with_timer
2021/12/17 12:26:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:26:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 12:26:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 12:26:26 Resuming app: bucket_op_with_timer
2021/12/17 12:26:28 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:26:28 Starting bucket ops to source bucket
2021/12/17 12:26:36 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 12:26:41 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 12:26:46 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 12:26:51 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/17 12:26:56 Waiting for dst bucket item count to get to: 15000 curr count: 10836
2021/12/17 12:27:01 Waiting for dst bucket item count to get to: 15000 curr count: 10836
2021/12/17 12:27:01 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/17 12:27:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:27:06 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 12:27:06 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 12:27:06 Pausing app: bucket_op_with_timer
2021/12/17 12:27:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:27:11 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 12:27:11 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 12:27:11 Resuming app: bucket_op_with_timer
2021/12/17 12:27:12 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:27:12 Starting bucket ops to source bucket
2021/12/17 12:27:20 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:27:25 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:27:30 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:27:35 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:27:40 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:27:45 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/17 12:27:50 Waiting for dst bucket item count to get to: 20000 curr count: 19034
2021/12/17 12:27:55 Waiting for dst bucket item count to get to: 20000 curr count: 19034
2021/12/17 12:27:55 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/17 12:27:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:28:00 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 12:28:00 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 12:28:00 Pausing app: bucket_op_with_timer
2021/12/17 12:28:00 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:28:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 12:28:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 12:28:05 Resuming app: bucket_op_with_timer
2021/12/17 12:28:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:28:06 Starting bucket ops to source bucket
2021/12/17 12:28:14 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:28:19 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:28:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:28:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:28:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/17 12:28:39 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/17 12:28:44 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/17 12:28:44 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/17 12:28:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:28:49 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/17 12:28:49 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/17 12:28:49 Pausing app: bucket_op_with_timer
2021/12/17 12:28:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:28:54 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/17 12:28:54 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/17 12:28:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:29:00 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 12:29:05 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 12:29:10 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 12:29:15 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/17 12:29:15 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/17 12:29:15 No eventing-con process running
2021/12/17 12:29:15 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/17 12:29:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:29:26 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 12:29:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:29:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:29:38 Waiting for dst bucket item count to get to: 0 curr count: 19028
2021/12/17 12:29:43 Waiting for dst bucket item count to get to: 0 curr count: 6528
2021/12/17 12:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.22s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/17 12:29:53 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 12:29:53 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/17 12:29:53 No eventing-con process running
2021/12/17 12:29:53 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/17 12:29:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:30:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:30:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:30:07 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:30:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/17 12:30:13 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 12:30:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 12:30:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 12:30:28 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/17 12:30:28 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/17 12:30:28 Starting bucket ops to source bucket
2021/12/17 12:30:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:30:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:30:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:30:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:30:57 Waiting for dst bucket item count to get to: 5000 curr count: 883
2021/12/17 12:31:02 Waiting for dst bucket item count to get to: 5000 curr count: 2842
2021/12/17 12:31:07 Waiting for dst bucket item count to get to: 5000 curr count: 4459
2021/12/17 12:31:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:31:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:31:07 Pausing app: bucket_op_with_timer
2021/12/17 12:31:07 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:31:12 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/17 12:31:12 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/17 12:31:12 Resuming app: bucket_op_with_timer
2021/12/17 12:31:14 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:31:14 Starting bucket ops to source bucket
2021/12/17 12:31:26 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:31:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:31:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:31:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:31:46 Waiting for dst bucket item count to get to: 10000 curr count: 6396
2021/12/17 12:31:51 Waiting for dst bucket item count to get to: 10000 curr count: 8896
2021/12/17 12:31:51 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 12:31:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:31:56 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 12:32:01 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 12:32:06 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 12:32:11 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/17 12:32:11 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/17 12:32:11 No eventing-con process running
2021/12/17 12:32:11 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/17 12:32:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:32:23 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 12:32:28 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:32:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:32:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:32:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:32:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.31s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/17 12:32:55 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 12:32:55 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/17 12:32:55 No eventing-con process running
2021/12/17 12:32:55 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/17 12:32:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:33:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:33:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:33:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:33:08 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:33:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/17 12:33:14 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 12:33:19 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 12:33:24 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 12:33:29 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/17 12:33:29 App: TestPauseResumeWithEventingReb got deployed
2021/12/17 12:33:29 Starting bucket ops to source bucket
2021/12/17 12:33:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:33:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:33:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:33:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:33:59 Waiting for dst bucket item count to get to: 5000 curr count: 1280
2021/12/17 12:34:04 Waiting for dst bucket item count to get to: 5000 curr count: 2560
2021/12/17 12:34:09 Waiting for dst bucket item count to get to: 5000 curr count: 3780
2021/12/17 12:34:09 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:34:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:34:09 Pausing app: bucket_op_with_timer
2021/12/17 12:34:09 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:34:14 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/17 12:34:14 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/17 12:34:14 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 12:34:14 Removing node(s): [] from the cluster
2021/12/17 12:34:19 Rebalance progress: 29.16666666666666
2021/12/17 12:34:24 Rebalance progress: 29.16666666666666
2021/12/17 12:34:29 Rebalance progress: 100
2021/12/17 12:34:29 Resuming app: bucket_op_with_timer
2021/12/17 12:34:30 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:34:30 Starting bucket ops to source bucket
2021/12/17 12:34:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:34:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:34:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:34:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:35:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/17 12:35:02 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 12:35:02 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 12:35:07 Rebalance progress: 29.16666666666666
2021/12/17 12:35:12 Rebalance progress: 29.16666666666666
2021/12/17 12:35:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/17 12:35:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:35:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:35:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:36:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:36:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:36:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:36:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:37:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:37:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:37:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 12:37:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:54} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9758e867b2b8a154bf750eafd8b37462
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
Producer node: 172.31.2.134:9300	Node UUID: 7d8216761bfe415a7b12ac448d29f953
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 5c9fb75688654c7a6a8535bcf4d725e9
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/17 12:38:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:38:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:38:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 12:38:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 12:38:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 12:38:27 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/17 12:38:27 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/17 12:38:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:38:28 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/17 12:38:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:38:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 12:38:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:38:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:38:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:38:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 12:38:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (371.69s)
=== RUN   TestBucketDeleteAfterPause
2021/12/17 12:39:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/17 12:39:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/17 12:39:06 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:39:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/17 12:39:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:39:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:39:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:39:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:39:20 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:39:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/17 12:39:26 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 12:39:31 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 12:39:36 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 12:39:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/17 12:39:41 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/17 12:39:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:39:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:39:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:39:51 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:39:51 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/17 12:39:51 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:39:56 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/17 12:39:56 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/17 12:39:56 handler_test.go:716: Deleting source bucket: default
2021/12/17 12:40:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/17 12:40:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/17 12:40:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/17 12:40:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/17 12:40:07 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:40:08 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/17 12:40:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:40:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 12:40:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:40:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:40:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4569
2021/12/17 12:40:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4569
2021/12/17 12:40:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.59s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/17 12:40:46 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 12:40:46 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/17 12:40:46 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:40:46 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/17 12:40:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:40:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:40:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:40:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:40:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:41:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/17 12:41:05 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 12:41:10 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 12:41:15 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 12:41:20 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/17 12:41:20 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/17 12:41:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:41:55 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/17 12:41:55 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:42:00 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/17 12:42:00 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/17 12:45:00 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/17 12:45:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/17 12:45:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/17 12:45:06 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 12:45:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:45:06 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/17 12:45:11 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 12:45:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:45:11 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3600 expected 5000
2021/12/17 12:45:16 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/17 12:45:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:45:16 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/17 12:45:16 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/17 12:45:16 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:45:26 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 12:45:31 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 12:45:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/17 12:45:36 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/17 12:45:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:45:36 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/17 12:45:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:45:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 12:45:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:45:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:45:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (318.67s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/17 12:46:04 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 12:46:04 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/17 12:46:04 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:46:04 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/17 12:46:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:46:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:46:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:46:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:46:18 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:46:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 12:46:24 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:46:29 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:46:34 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:46:39 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:46:39 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/17 12:46:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:46:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:46:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 270
2021/12/17 12:46:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2770
2021/12/17 12:46:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:46:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:46:58 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/17 12:46:58 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:47:28 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/17 12:47:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 12:47:34 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:47:39 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:47:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:47:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/17 12:47:49 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/17 12:47:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:48:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:48:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:48:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:48:02 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/17 12:48:02 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:48:12 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 12:48:17 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 12:48:22 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/17 12:48:22 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/17 12:48:22 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:48:22 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/17 12:48:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:48:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:48:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:48:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (160.72s)
=== RUN   TestCPPWorkerCleanup
2021/12/17 12:48:45 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 12:48:45 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/17 12:48:45 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:48:45 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/17 12:48:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:48:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:48:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:48:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:48:58 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:48:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/17 12:48:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:49:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:49:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:49:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:49:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:49:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 12:49:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:49:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:49:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:49:32 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 12:49:37 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 12:49:42 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 12:49:47 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/17 12:49:47 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/17 12:49:47 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:49:47 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/17 12:49:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:49:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4236
2021/12/17 12:50:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1736
2021/12/17 12:50:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:50:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:50:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:50:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.75s)
=== RUN   TestWithUserXattrs
2021/12/17 12:50:55 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 12:50:55 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/17 12:50:55 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:50:55 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/17 12:50:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:51:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:51:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:51:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:51:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:51:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:51:12 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:51:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/17 12:51:18 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 12:51:23 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 12:51:28 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 12:51:33 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/17 12:51:33 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/17 12:51:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:51:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:51:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2370
2021/12/17 12:51:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4870
2021/12/17 12:51:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:51:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:51:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 12:51:58 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 12:52:03 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 12:52:08 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 12:52:13 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/17 12:52:13 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/17 12:52:13 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:52:13 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/17 12:52:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:52:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 12:52:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 12:52:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:52:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:52:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4546
2021/12/17 12:52:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4546
2021/12/17 12:52:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (120.91s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/17 12:52:56 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 12:52:56 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/17 12:52:56 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:52:56 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/17 12:52:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:53:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:53:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:53:09 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:53:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/17 12:53:15 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 12:53:20 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 12:53:25 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 12:53:30 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/17 12:53:30 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/17 12:53:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:53:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:53:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 12:53:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 12:53:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:53:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:53:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 12:53:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:54:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:54:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 12:54:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 12:54:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:54:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 12:54:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16123
2021/12/17 12:54:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16123
2021/12/17 12:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/17 12:54:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 12:54:46 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 12:54:51 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 12:54:56 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 12:55:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 12:55:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 12:55:06 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/17 12:55:06 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:55:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/17 12:55:11 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/17 12:55:11 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:55:11 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/17 12:55:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:55:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 12:55:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/17 12:55:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:55:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:55:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 24056
2021/12/17 12:55:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11556
2021/12/17 12:55:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.97s)
=== RUN   TestMetaBucketDelete
2021/12/17 12:55:54 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/17 12:55:54 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/17 12:55:54 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:55:54 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/17 12:55:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:56:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:56:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:56:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:56:07 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:56:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 12:56:13 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 12:56:18 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 12:56:23 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 12:56:28 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/17 12:56:28 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/17 12:56:28 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/17 12:56:30 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/17 12:56:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/17 12:56:35 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/17 12:56:50 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/17 12:56:50 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/17 12:56:50 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:56:50 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/17 12:56:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:56:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:56:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:57:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (79.05s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/17 12:57:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 12:57:13 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 12:57:13 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:57:13 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/17 12:57:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:57:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:57:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:57:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:57:26 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:57:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 12:57:46 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/17 12:57:47 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/17 12:58:02 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 12:58:02 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 12:58:17 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 12:58:17 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 12:58:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:58:17 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/17 12:58:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:58:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:58:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:58:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.07s)
=== RUN   TestSourceBucketDelete
2021/12/17 12:58:40 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/17 12:58:40 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/17 12:58:40 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:58:40 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/17 12:58:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:58:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:58:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:58:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:58:53 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/17 12:58:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/17 12:58:59 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 12:59:04 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 12:59:09 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 12:59:14 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/17 12:59:14 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/17 12:59:14 handler_test.go:945: Deleting source bucket: default
2021/12/17 12:59:16 handler_test.go:947: Deleted source bucket: default
2021/12/17 12:59:21 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/17 12:59:21 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/17 12:59:36 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/17 12:59:36 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/17 12:59:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:59:36 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/17 12:59:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:59:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 12:59:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 12:59:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.10s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/17 12:59:59 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/17 12:59:59 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/17 12:59:59 eventingSetup.go:46: No eventing-con process running
2021/12/17 12:59:59 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/17 13:00:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:00:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:00:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:00:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:00:12 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:00:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 13:00:32 handler_test.go:966: Deleting source bucket: default
2021/12/17 13:00:33 handler_test.go:968: Deleted source bucket: default
2021/12/17 13:00:48 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/17 13:00:48 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/17 13:01:03 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/17 13:01:03 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/17 13:01:03 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:01:04 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/17 13:01:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:01:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:01:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:01:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.68s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/17 13:01:27 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 13:01:27 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 13:01:27 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:01:27 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/17 13:01:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:01:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:01:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:01:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:01:40 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:01:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 13:02:00 handler_test.go:988: Deleting source bucket: default
2021/12/17 13:02:01 handler_test.go:990: Deleted source bucket: default
2021/12/17 13:02:01 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/17 13:02:02 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/17 13:02:17 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 13:02:17 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 13:02:32 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/17 13:02:32 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/17 13:02:32 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:02:32 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/17 13:02:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:02:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:02:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:02:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.12s)
=== RUN   TestUndeployDuringBootstrap
2021/12/17 13:02:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 13:02:55 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/17 13:02:55 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:02:55 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/17 13:02:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:03:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:03:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:03:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:03:08 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:03:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 13:03:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 13:03:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:03:21 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:03:21 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/17 13:03:21 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/17 13:03:21 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/17 13:03:26 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/17 13:03:31 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/17 13:03:36 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 13:03:41 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 13:03:46 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 13:03:46 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/17 13:03:46 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:04:11 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/17 13:04:11 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/17 13:04:11 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:04:11 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/17 13:04:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:04:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 13:04:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 13:04:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:04:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:04:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3642
2021/12/17 13:04:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1142
2021/12/17 13:04:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.47s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/17 13:04:49 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:04:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/17 13:04:55 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 13:05:00 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 13:05:05 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 13:05:10 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/17 13:05:11 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/17 13:05:11 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/17 13:05:11 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 13:05:11 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/17 13:05:16 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/17 13:05:16 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/17 13:05:16 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:05:16 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/17 13:05:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:05:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:05:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:05:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.68s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/17 13:05:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 13:05:39 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/17 13:05:39 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:05:39 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/17 13:05:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:05:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:05:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:05:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:05:52 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:05:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/17 13:05:59 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 13:06:04 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 13:06:09 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 13:06:14 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/17 13:06:14 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/17 13:06:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 13:06:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:06:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 13:06:54 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 13:06:59 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 13:07:04 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 13:07:04 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/17 13:07:04 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:08:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/17 13:08:49 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/17 13:08:49 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:08:49 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/17 13:08:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:09:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4098
2021/12/17 13:09:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4098
2021/12/17 13:09:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:09:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:09:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 13:09:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.12s)
=== RUN   TestUndeployWithKVFailover
2021/12/17 13:09:27 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/17 13:09:27 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/17 13:09:27 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:09:27 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/17 13:09:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:09:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:09:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:09:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:09:40 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:09:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/17 13:09:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 13:09:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:09:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:09:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:10:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:10:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:10:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:10:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1638
2021/12/17 13:10:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1638
2021/12/17 13:10:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 13:10:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 13:10:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 13:10:29 Rebalance progress: 29.16666666666666
2021/12/17 13:10:34 Rebalance progress: 29.16666666666666
2021/12/17 13:10:39 Rebalance progress: 100
2021/12/17 13:10:39 Failing over: [127.0.0.1:19003]
2021/12/17 13:10:49 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/17 13:11:49 Removing node(s): [] from the cluster
2021/12/17 13:11:54 Rebalance progress: 29.16666666666666
2021/12/17 13:11:59 Rebalance progress: 29.16666666666666
2021/12/17 13:12:04 Rebalance progress: 100
2021/12/17 13:12:09 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/17 13:12:09 App: TestUndeployWithKVFailover got un-deployed
2021/12/17 13:12:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:12:14 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/17 13:12:14 App: TestUndeployWithKVFailover got un-deployed
2021/12/17 13:12:14 No eventing-con process running
2021/12/17 13:12:14 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/17 13:12:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:12:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 13:12:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:12:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:12:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 13:12:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.31s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/17 13:12:47 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 13:12:47 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/17 13:12:47 No eventing-con process running
2021/12/17 13:12:47 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/17 13:12:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:12:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:12:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:13:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:13:01 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:13:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/17 13:13:07 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 13:13:12 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 13:13:17 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 13:13:22 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/17 13:13:22 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/17 13:13:22 Starting bucket ops to source bucket
2021/12/17 13:13:43 Waiting for dst bucket item count to get to: 20000 curr count: 8206
2021/12/17 13:13:48 Waiting for dst bucket item count to get to: 20000 curr count: 19829
2021/12/17 13:13:53 Waiting for dst bucket item count to get to: 20000 curr count: 19829
2021/12/17 13:13:53 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/17 13:13:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:13:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:14:05 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/17 13:14:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:14:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:14:17 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/17 13:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:14:17 Starting bucket ops to source bucket
2021/12/17 13:14:30 Waiting for dst bucket item count to get to: 10000 curr count: 1287
2021/12/17 13:14:35 Waiting for dst bucket item count to get to: 10000 curr count: 6287
2021/12/17 13:14:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 13:14:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:14:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:14:40 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 13:14:45 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 13:14:50 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 13:14:55 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/17 13:14:55 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/17 13:14:55 No eventing-con process running
2021/12/17 13:14:55 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/17 13:14:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:15:07 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 13:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:15:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:15:18 Waiting for dst bucket item count to get to: 0 curr count: 8246
2021/12/17 13:15:23 Waiting for dst bucket item count to get to: 0 curr count: 3246
2021/12/17 13:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (165.97s)
=== RUN   TestTimerOverwrite
2021/12/17 13:15:33 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 13:15:33 App: TestTimerOverwrite got un-deployed
2021/12/17 13:15:33 No eventing-con process running
2021/12/17 13:15:33 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/17 13:15:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:15:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:15:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:15:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:15:47 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:15:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/17 13:15:53 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 13:15:58 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 13:16:03 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 13:16:08 Waiting for app: TestTimerOverwrite to get deployed
2021/12/17 13:16:08 App: TestTimerOverwrite got deployed
2021/12/17 13:16:18 Starting bucket ops to source bucket
2021/12/17 13:16:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:16:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:16:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:16:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:16:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:16:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:16:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:17:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:17:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:17:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:17:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:17:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:17:27 Waiting for dst bucket item count to get to: 5000 curr count: 2564
2021/12/17 13:17:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 13:17:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:17:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:17:42 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 13:17:47 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 13:17:52 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 13:17:57 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/17 13:17:57 App: TestTimerOverwrite got un-deployed
2021/12/17 13:17:57 No eventing-con process running
2021/12/17 13:17:57 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/17 13:17:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:18:08 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 13:18:13 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 13:18:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:18:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:18:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 13:18:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (181.96s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/17 13:18:30 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 13:18:31 Removing node(s): [] from the cluster
2021/12/17 13:18:36 Rebalance progress: 100
2021/12/17 13:18:36 Failing over: [127.0.0.1:19003]
2021/12/17 13:18:41 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/17 13:18:41 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/17 13:18:41 No eventing-con process running
2021/12/17 13:18:41 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/17 13:18:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:18:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:18:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:18:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:18:54 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:18:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/17 13:18:55 Starting bucket ops to source bucket
2021/12/17 13:19:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:19:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:19:05 Context: Post to set retry counter, response code: 200 dump: 
2021/12/17 13:19:05 set retry counter, response &{[] }
2021/12/17 13:20:10 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/17 13:20:10 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/17 13:20:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:20:10 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/17 13:20:15 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/17 13:20:15 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/17 13:20:15 No eventing-con process running
2021/12/17 13:20:15 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/17 13:20:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:20:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 13:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:20:28 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:20:38 Waiting for dst bucket item count to get to: 0 curr count: 3092
2021/12/17 13:20:43 Waiting for dst bucket item count to get to: 0 curr count: 592
2021/12/17 13:20:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:20:43 Removing node(s): [] from the cluster
2021/12/17 13:20:48 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (143.04s)
=== RUN   TestOnUpdateSrcMutation
2021/12/17 13:20:58 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 13:20:58 App: TestOnUpdateSrcMutation got un-deployed
2021/12/17 13:20:58 No eventing-con process running
2021/12/17 13:20:58 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/17 13:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:21:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:21:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:21:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:21:12 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:21:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/17 13:21:13 Starting bucket ops to source bucket
2021/12/17 13:21:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 13:21:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:21:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:21:40 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 13:21:45 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 13:21:50 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 13:21:55 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/17 13:21:55 App: TestOnUpdateSrcMutation got un-deployed
2021/12/17 13:21:55 No eventing-con process running
2021/12/17 13:21:55 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/17 13:21:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:22:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 13:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:22:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:22:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.51s)
=== RUN   TestOnDeleteSrcMutation
2021/12/17 13:22:23 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 13:22:23 App: TestOnDeleteSrcMutation got un-deployed
2021/12/17 13:22:23 No eventing-con process running
2021/12/17 13:22:23 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/17 13:22:24 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:22:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:22:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:22:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:22:36 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:22:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/17 13:22:37 Starting bucket ops to source bucket
2021/12/17 13:23:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 13:23:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:23:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:23:11 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 13:23:16 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 13:23:21 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 13:23:26 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/17 13:23:26 App: TestOnDeleteSrcMutation got un-deployed
2021/12/17 13:23:26 No eventing-con process running
2021/12/17 13:23:26 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/17 13:23:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:23:37 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 13:23:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:23:39 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.33s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/17 13:23:54 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 13:23:54 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/17 13:23:54 No eventing-con process running
2021/12/17 13:23:54 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/17 13:23:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:24:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:24:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:24:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:24:07 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:24:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 13:24:09 Starting bucket ops to source bucket
2021/12/17 13:24:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 13:24:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:24:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:24:51 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 13:24:56 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 13:25:01 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 13:25:06 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/17 13:25:06 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/17 13:25:06 No eventing-con process running
2021/12/17 13:25:06 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/17 13:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:25:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 13:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:25:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:25:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.54s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/17 13:25:34 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 13:25:34 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/17 13:25:34 No eventing-con process running
2021/12/17 13:25:34 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/17 13:25:35 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:25:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:25:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:25:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:25:47 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:25:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/17 13:25:48 Starting bucket ops to source bucket
2021/12/17 13:26:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 13:26:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:26:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:26:42 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 13:26:47 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 13:26:52 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 13:26:57 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/17 13:26:57 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/17 13:26:57 No eventing-con process running
2021/12/17 13:26:57 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/17 13:26:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:27:08 Waiting for dst bucket item count to get to: 0 curr count: 4644
2021/12/17 13:27:13 Waiting for dst bucket item count to get to: 0 curr count: 2144
2021/12/17 13:27:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:27:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:27:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (116.31s)
=== RUN   TestInterHandlerRecursion
2021/12/17 13:27:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:27:30 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 13:27:30 No eventing-con process running
2021/12/17 13:27:30 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/17 13:27:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:27:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:27:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:27:48 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:27:48 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 13:27:48 No eventing-con process running
2021/12/17 13:27:48 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/17 13:27:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:27:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:27:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:28:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:28:01 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:28:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/17 13:28:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:28:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:28:12 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:28:17 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:28:22 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:28:22 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/17 13:28:23 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:28:23 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/17 13:28:23 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/17 13:28:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:28:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:28:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:28:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:28:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:28:43 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 13:28:43 No eventing-con process running
2021/12/17 13:28:43 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/17 13:28:44 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:28:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:28:51 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:29:01 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:29:01 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 13:29:01 No eventing-con process running
2021/12/17 13:29:01 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/17 13:29:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:29:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:29:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.80s)
=== RUN   TestInterBucketRecursion
2021/12/17 13:29:19 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:29:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 13:29:20 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:29:20 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:29:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/17 13:29:25 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 13:29:30 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 13:29:35 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 13:29:40 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/17 13:29:40 App: TestInterBucketRecursion_function1 got deployed
2021/12/17 13:29:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:29:45 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:29:50 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:29:55 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:30:00 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:30:00 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/17 13:30:00 No eventing-con process running
2021/12/17 13:30:00 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/17 13:30:01 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:30:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:30:18 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/17 13:30:18 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/17 13:30:18 No eventing-con process running
2021/12/17 13:30:18 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/17 13:30:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:30:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:30:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:30:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.56s)
=== RUN   TestLargeHandler
2021/12/17 13:30:31 Changed force_compress value to false
2021/12/17 13:30:31 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:30:32 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/17 13:30:32 Changed force_compress value to true
2021/12/17 13:30:32 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:30:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/17 13:30:38 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 13:30:43 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 13:30:48 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 13:30:53 Waiting for app: TestLargeHandler_function to get deployed
2021/12/17 13:30:53 App: TestLargeHandler_function got deployed
2021/12/17 13:30:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:30:59 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 13:31:04 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 13:31:09 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 13:31:14 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/17 13:31:14 App: TestLargeHandler_function got un-deployed
2021/12/17 13:31:14 No eventing-con process running
2021/12/17 13:31:14 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/17 13:31:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:31:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:31:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:31:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.40s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/17 13:31:37 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:31:37 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 13:31:37 No eventing-con process running
2021/12/17 13:31:37 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/17 13:31:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:31:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:31:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:31:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:31:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:31:55 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 13:31:55 No eventing-con process running
2021/12/17 13:31:55 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/17 13:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:32:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:32:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:32:08 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:32:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/17 13:32:09 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:32:14 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:32:19 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:32:24 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:32:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/17 13:32:29 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/17 13:32:29 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:32:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/17 13:32:31 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:32:36 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 13:32:41 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 13:32:46 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 13:32:51 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/17 13:32:51 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/17 13:32:51 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/17 13:32:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:32:56 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:33:01 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:33:06 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:33:11 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/17 13:33:11 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/17 13:33:11 No eventing-con process running
2021/12/17 13:33:11 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/17 13:33:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:33:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:33:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:33:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:33:30 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:33:35 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:33:40 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:33:45 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/17 13:33:45 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/17 13:33:45 No eventing-con process running
2021/12/17 13:33:45 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/17 13:33:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:33:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:33:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:33:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.77s)
=== RUN   TestAllowInterBucketRecursion
2021/12/17 13:34:04 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:34:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 13:34:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:34:10 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 13:34:15 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 13:34:20 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 13:34:25 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/17 13:34:25 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/17 13:34:25 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:34:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/17 13:34:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:34:31 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 13:34:36 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 13:34:41 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 13:34:46 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/17 13:34:46 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/17 13:34:46 TestAllowInterBucketRecursion_function2 is deployed
2021/12/17 13:34:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:34:51 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:34:56 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:35:01 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:35:06 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/17 13:35:06 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/17 13:35:06 No eventing-con process running
2021/12/17 13:35:06 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/17 13:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:35:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:35:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:35:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:35:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:35:26 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 13:35:31 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 13:35:36 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 13:35:41 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/17 13:35:41 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/17 13:35:41 No eventing-con process running
2021/12/17 13:35:41 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/17 13:35:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:35:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:35:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:35:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.45s)
=== RUN   TestInterBucketRecursion2
2021/12/17 13:36:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:36:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:36:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:36:23 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:36:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 13:36:29 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 13:36:34 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 13:36:39 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 13:36:44 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/17 13:36:44 App: TestInterBucketRecursion2_function1 got deployed
2021/12/17 13:36:44 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:36:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/17 13:36:51 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 13:36:56 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 13:37:01 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 13:37:06 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/17 13:37:06 App: TestInterBucketRecursion2_function2 got deployed
2021/12/17 13:37:06 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:37:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/17 13:37:12 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 13:37:17 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 13:37:22 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 13:37:27 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/17 13:37:27 App: TestInterBucketRecursion2_function3 got deployed
2021/12/17 13:37:27 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:37:27 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/17 13:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:37:32 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 13:37:37 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 13:37:42 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 13:37:47 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/17 13:37:47 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/17 13:37:47 No eventing-con process running
2021/12/17 13:37:47 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/17 13:37:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:37:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:37:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:38:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:38:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:38:05 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 13:38:10 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 13:38:15 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 13:38:20 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/17 13:38:20 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/17 13:38:20 No eventing-con process running
2021/12/17 13:38:20 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/17 13:38:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:38:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:38:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:38:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:38:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:38:39 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 13:38:44 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 13:38:49 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 13:38:54 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/17 13:38:54 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/17 13:38:54 No eventing-con process running
2021/12/17 13:38:54 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/17 13:38:55 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:39:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:39:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:39:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:39:12 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/17 13:39:12 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/17 13:39:12 No eventing-con process running
2021/12/17 13:39:12 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/17 13:39:13 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:39:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:39:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:39:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.10s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/17 13:39:32 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 13:39:37 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/17 13:39:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:39:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:40:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:40:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:40:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:40:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:41:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:41:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:41:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/17 13:41:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/17 13:42:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:42:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 13:42:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 13:42:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/17 13:42:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:42:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 13:42:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 13:42:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/17 13:42:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:42:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/17 13:42:26 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/17 13:42:26 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:42:26 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/17 13:42:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:42:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:42:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:42:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:42:44 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:42:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 13:42:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 13:42:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 13:43:00 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 13:43:05 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/17 13:43:05 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9758e867b2b8a154bf750eafd8b37462
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 7d8216761bfe415a7b12ac448d29f953
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 5c9fb75688654c7a6a8535bcf4d725e9
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/17 13:43:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:43:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 13:43:11 Rebalance progress: 29.16666666666666
2021/12/17 13:43:16 Rebalance progress: 29.16666666666666
2021/12/17 13:43:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9758e867b2b8a154bf750eafd8b37462
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 7d8216761bfe415a7b12ac448d29f953
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 5c9fb75688654c7a6a8535bcf4d725e9
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]


2021/12/17 13:43:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:43:21 handler_test.go:1674: Deleting source bucket: default
2021/12/17 13:43:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 13:43:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 13:43:28 Rebalance progress: 29.16666666666666
2021/12/17 13:43:33 Rebalance progress: 29.16666666666666
2021/12/17 13:43:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 893
	vb list dump: [0-4, 6-9, 11-30, 32-38, 40-53, 55-62, 65-66, 68-79, 83-85, 87-92, 94-96, 98-111, 113-115, 117-124, 126-128, 130-135, 137-138, 140-148, 150, 152-156, 158, 160-163, 165-174, 176-180, 182-190, 192-209, 211-214, 216-228, 230-251, 253-264, 266, 269-272, 274-287, 289-290, 292-298, 302-304, 306-314, 316-320, 323-332, 334-341, 344, 346-349, 351-352, 354-361, 363-366, 368-381, 383-391, 393-397, 399-405, 407-417, 419-420, 422-424, 426-430, 432, 434-438, 440-443, 445, 447-454, 457-461, 464, 469-472, 474-478, 480-488, 490-491, 493-499, 501, 503-509, 511-586, 588-591, 593-604, 606-609, 611, 613-615, 617-629, 631-637, 639-640, 643-674, 676-678, 680-688, 690-696, 698-727, 729-737, 739-743, 746-753, 756-763, 765-766, 768-777, 779-780, 782-792, 794-802, 805, 808-830, 833-836, 838-869, 871-872, 874-877, 879-881, 883-886, 888-899, 901-919, 922-933, 935, 937-939, 941-948, 950-958, 960-966, 968-982, 984-988, 990-1000, 1002-1006, 1008-1011, 1013-1023]
	stream status: stopped
	len: 131
	vb list dump: [5, 10, 31, 39, 54, 63-64, 67, 80-82, 86, 93, 97, 112, 116, 125, 129, 136, 139, 149, 151, 157, 159, 164, 175, 181, 191, 210, 215, 229, 252, 265, 267-268, 273, 288, 291, 299-301, 305, 315, 321-322, 333, 342-343, 345, 350, 353, 362, 367, 382, 392, 398, 406, 418, 421, 425, 431, 433, 439, 444, 446, 455-456, 462-463, 465-468, 473, 479, 489, 492, 500, 502, 510, 587, 592, 605, 610, 612, 616, 630, 638, 641-642, 675, 679, 689, 697, 728, 738, 744-745, 754-755, 764, 767, 778, 781, 793, 803-804, 806-807, 831-832, 837, 870, 873, 878, 882, 887, 900, 920-921, 934, 936, 940, 949, 959, 967, 983, 989, 1001, 1007, 1012]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 9758e867b2b8a154bf750eafd8b37462
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [114-115, 117-124, 126-128, 130-135, 137-138, 140-148, 150, 152-156, 158, 160-163, 165-174, 176-180, 182-190, 192-209, 211-214, 216-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 98
	v: [228, 230-251, 253-264, 266, 269-272, 274-287, 289-290, 292-298, 302-304, 306-314, 316-320, 323-332, 334-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [0-4, 6-9, 11-30, 32-38, 40-53, 55-62, 65-66, 68-79, 83-85, 87-92, 94-96, 98-111, 113]
Producer node: 172.31.2.134:9300	Node UUID: 7d8216761bfe415a7b12ac448d29f953
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [911-919, 922-933, 935, 937-939, 941-948, 950-958, 960-966, 968-982, 984-988, 990-1000, 1002-1006, 1008-1011, 1013-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-688, 690-696, 698-727, 729-737, 739-743, 746-753, 756-763, 765-766, 768-777, 779-780, 782-792, 794-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-802, 805, 808-830, 833-836, 838-869, 871-872, 874-877, 879-881, 883-886, 888-899, 901-910]
Producer node: 127.0.0.1:9303	Node UUID: 5c9fb75688654c7a6a8535bcf4d725e9
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [457-461, 464, 469-472, 474-478, 480-488, 490-491, 493-499, 501, 503-509, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-586, 588-591, 593-604, 606-609, 611, 613-615, 617-629, 631-637, 639-640, 643-674, 676-678, 680-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 94
	v: [344, 346-349, 351-352, 354-361, 363-366, 368-381, 383-391, 393-397, 399-405, 407-417, 419-420, 422-424, 426-430, 432, 434-438, 440-443, 445, 447-454]


2021/12/17 13:43:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:43:43 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/17 13:43:43 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/17 13:43:48 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/17 13:43:48 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/17 13:43:48 eventingSetup.go:46: No eventing-con process running
2021/12/17 13:43:48 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/17 13:43:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:43:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:43:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:44:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:44:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/17 13:44:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 13:44:07 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (282.66s)
=== RUN   TestN1QLRecursion
2021/12/17 13:44:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 13:44:10 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/17 13:44:10 Removing node(s): [] from the cluster
2021/12/17 13:44:15 Rebalance progress: 100
2021/12/17 13:44:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:44:27 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:44:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:44:38 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:44:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 13:44:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:44:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 13:44:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 13:44:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 13:44:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 13:45:04 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 13:45:09 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/17 13:45:09 App: TestN1QLRecursion_function1 got deployed
2021/12/17 13:45:09 TestN1QLRecursion_function1 is deployed
2021/12/17 13:45:09 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:45:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/17 13:45:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:45:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 13:45:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 13:45:25 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 13:45:30 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/17 13:45:30 App: TestN1QLRecursion_function3 got deployed
2021/12/17 13:45:30 TestN1QLRecursion_function3 is deployed
2021/12/17 13:45:30 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:45:30 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/17 13:45:30 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/17 13:45:30 Success: TestN1QLRecursion_function2 is not deployed
2021/12/17 13:45:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:45:35 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 13:45:40 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 13:45:45 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 13:45:50 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/17 13:45:50 App: TestN1QLRecursion_function1 got un-deployed
2021/12/17 13:45:50 No eventing-con process running
2021/12/17 13:45:50 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/17 13:45:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:45:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:45:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:46:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:46:09 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 13:46:14 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 13:46:19 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 13:46:24 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/17 13:46:24 App: TestN1QLRecursion_function3 got un-deployed
2021/12/17 13:46:24 No eventing-con process running
2021/12/17 13:46:24 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/17 13:46:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:46:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:46:33 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:46:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:46:39 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 13:46:45 Rebalance progress: 100
--- PASS: TestN1QLRecursion (154.81s)
=== RUN   TestN1QLAllowRecursion
2021/12/17 13:46:45 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 13:46:45 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/17 13:46:45 Removing node(s): [] from the cluster
2021/12/17 13:46:50 Rebalance progress: 100
2021/12/17 13:46:56 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:47:02 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:47:07 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:47:13 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:47:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/17 13:47:14 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:47:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 13:47:24 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 13:47:29 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 13:47:34 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/17 13:47:34 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/17 13:47:34 TestN1QLAllowRecursion_function1 is deployed
2021/12/17 13:47:34 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:47:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/17 13:47:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:47:40 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 13:47:45 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 13:47:50 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 13:47:55 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/17 13:47:55 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/17 13:47:55 TestN1QLAllowRecursion_function3 is deployed
2021/12/17 13:47:55 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:47:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/17 13:47:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/17 13:48:01 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 13:48:06 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 13:48:11 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 13:48:16 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/17 13:48:16 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/17 13:48:16 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/17 13:48:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:48:21 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 13:48:26 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 13:48:31 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 13:48:36 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/17 13:48:36 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/17 13:48:36 No eventing-con process running
2021/12/17 13:48:37 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/17 13:48:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:48:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:48:45 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:48:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:48:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:48:55 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 13:49:00 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 13:49:05 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 13:49:10 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/17 13:49:10 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/17 13:49:10 No eventing-con process running
2021/12/17 13:49:10 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/17 13:49:12 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:49:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:49:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:49:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:49:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:49:29 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 13:49:34 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 13:49:39 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 13:49:44 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/17 13:49:44 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/17 13:49:44 No eventing-con process running
2021/12/17 13:49:44 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/17 13:49:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:49:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:49:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:49:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:50:00 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 13:50:05 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.08s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/17 13:50:05 Sleeping for some time
2021/12/17 13:50:15 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 13:50:15 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/17 13:50:15 No eventing-con process running
2021/12/17 13:50:15 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/17 13:50:16 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:50:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:50:23 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:50:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:50:28 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:50:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 13:50:29 Starting bucket ops to source bucket
2021/12/17 13:51:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:51:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 13:51:29 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/17 13:51:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 13:51:29 Starting bucket ops to source bucket
2021/12/17 13:51:30 Starting bucket ops to source bucket
2021/12/17 13:52:36 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/17 13:52:41 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/17 13:52:41 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/17 13:52:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:52:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:52:46 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 13:52:51 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 13:52:56 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 13:53:01 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/17 13:53:01 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/17 13:53:01 No eventing-con process running
2021/12/17 13:53:01 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/17 13:53:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:53:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 13:53:18 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/17 13:53:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:53:20 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:53:30 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/17 13:53:35 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/17 13:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (209.97s)
=== RUN   TestTimerOverWriteSameReference
2021/12/17 13:53:40 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 13:53:40 App: TestTimerOverWriteSameReference got un-deployed
2021/12/17 13:53:40 No eventing-con process running
2021/12/17 13:53:40 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/17 13:53:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:53:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:53:53 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:53:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/17 13:53:59 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 13:54:04 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 13:54:09 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 13:54:15 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/17 13:54:15 App: TestTimerOverWriteSameReference got deployed
2021/12/17 13:54:25 Starting bucket ops to source bucket
2021/12/17 13:55:26 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/17 13:55:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:55:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:55:41 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 13:55:46 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 13:55:51 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 13:55:56 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/17 13:55:56 App: TestTimerOverWriteSameReference got un-deployed
2021/12/17 13:55:56 No eventing-con process running
2021/12/17 13:55:56 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/17 13:55:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:56:08 Waiting for dst bucket item count to get to: 0 curr count: 585
2021/12/17 13:56:13 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/17 13:56:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:56:15 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:56:25 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/17 13:56:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (170.02s)
=== RUN   TestCancelTimerBucketop
2021/12/17 13:56:30 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 13:56:30 App: TestCancelTimerBucketop got un-deployed
2021/12/17 13:56:30 No eventing-con process running
2021/12/17 13:56:30 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/17 13:56:32 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:56:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:56:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:56:43 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/17 13:56:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/17 13:56:49 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 13:56:55 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 13:57:00 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 13:57:05 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/17 13:57:05 App: TestCancelTimerBucketop got deployed
2021/12/17 13:57:05 Starting bucket ops to source bucket
2021/12/17 13:57:12 Starting bucket ops to source bucket
2021/12/17 13:58:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 13:58:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:58:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 13:58:19 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 13:58:24 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 13:58:29 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 13:58:34 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/17 13:58:34 App: TestCancelTimerBucketop got un-deployed
2021/12/17 13:58:34 No eventing-con process running
2021/12/17 13:58:34 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/17 13:58:36 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:58:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 13:58:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:58:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 13:58:58 Waiting for dst bucket item count to get to: 0 curr count: 610
2021/12/17 13:59:03 Waiting for dst bucket item count to get to: 0 curr count: 110
2021/12/17 13:59:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.14s)
=== RUN   TestJSExpiryDate
2021/12/17 13:59:08 Starting bucket ops to source bucket
2021/12/17 13:59:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 13:59:26 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/17 13:59:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/17 13:59:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 13:59:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 13:59:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:00:02 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 14:00:07 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 14:00:12 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 14:00:17 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/17 14:00:17 App: TestJSExpiryDate got un-deployed
2021/12/17 14:00:17 No eventing-con process running
2021/12/17 14:00:17 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/17 14:00:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:00:26 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:00:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.56s)
=== RUN   TestBinaryDoc
2021/12/17 14:00:36 Starting bucket ops to source bucket
2021/12/17 14:00:36 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:00:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/17 14:00:38 Starting bucket ops to source bucket
2021/12/17 14:00:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:00:38 Deleting source bucket docs now
2021/12/17 14:00:38 Starting bucket ops to source bucket
2021/12/17 14:01:15 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/17 14:01:20 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/17 14:01:25 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/17 14:01:25 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/17 14:01:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:01:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:01:30 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 14:01:35 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 14:01:40 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 14:01:45 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/17 14:01:45 App: TestBinaryDoc got un-deployed
2021/12/17 14:01:45 No eventing-con process running
2021/12/17 14:01:45 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/17 14:01:47 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:01:57 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/17 14:01:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:01:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:02:09 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/17 14:02:14 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/17 14:02:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.38s)
=== RUN   TestConstantBindings
2021/12/17 14:02:14 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:02:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/17 14:02:15 Starting bucket ops to source bucket
2021/12/17 14:02:21 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/17 14:02:26 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/17 14:02:31 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/17 14:02:36 Waiting for dst bucket item count to get to: 6000 curr count: 963
2021/12/17 14:02:41 Waiting for dst bucket item count to get to: 6000 curr count: 3963
2021/12/17 14:02:41 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/17 14:02:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:02:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:02:46 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 14:02:51 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 14:02:56 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 14:03:01 Waiting for app: TestConstantBindings to get un-deployed
2021/12/17 14:03:01 App: TestConstantBindings got un-deployed
2021/12/17 14:03:01 No eventing-con process running
2021/12/17 14:03:01 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/17 14:03:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:03:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:03:18 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 14:03:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:03:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:03:29 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/17 14:03:34 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/17 14:03:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.66s)
=== RUN   TestCodeInsight
2021/12/17 14:03:39 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 14:03:39 App: TestCodeInsight got un-deployed
2021/12/17 14:03:39 No eventing-con process running
2021/12/17 14:03:39 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/17 14:03:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:03:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:03:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:03:53 Starting bucket ops to source bucket
2021/12/17 14:03:54 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:03:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/17 14:04:00 Waiting for app: TestCodeInsight to get deployed
2021/12/17 14:04:05 Waiting for app: TestCodeInsight to get deployed
2021/12/17 14:04:10 Waiting for app: TestCodeInsight to get deployed
2021/12/17 14:04:15 Waiting for app: TestCodeInsight to get deployed
2021/12/17 14:04:15 App: TestCodeInsight got deployed
2021/12/17 14:04:20 Waiting for dst bucket item count to get to: 1000 curr count: 717
2021/12/17 14:04:20 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 14:04:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:04:25 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 14:04:30 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 14:04:35 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 14:04:40 Waiting for app: TestCodeInsight to get un-deployed
2021/12/17 14:04:40 App: TestCodeInsight got un-deployed
2021/12/17 14:04:40 No eventing-con process running
2021/12/17 14:04:40 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/17 14:04:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:04:52 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:04:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:04:59 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:05:09 Waiting for dst bucket item count to get to: 0 curr count: 935
2021/12/17 14:05:14 Waiting for dst bucket item count to get to: 0 curr count: 435
2021/12/17 14:05:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.36s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.06s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12993.334s

Functional tests took 3 hours, 36 minutes and 39 seconds.

N1ql tests

2021/12/17 14:05:23 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/17 14:05:33 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/17 14:05:33 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/17 14:05:33 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 14:05:34 Removing node(s): [] from the cluster
2021/12/17 14:05:39 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/17 14:05:44 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/17 14:05:44 App: TestRecursiveMutationN1QL got un-deployed
2021/12/17 14:05:44 No eventing-con process running
2021/12/17 14:05:44 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/17 14:05:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:05:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:05:52 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:05:57 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:05:57 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.58s)
=== RUN   TestFlexReset1
2021/12/17 14:06:02 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 14:06:02 App: TestFlexReset1 got un-deployed
2021/12/17 14:06:03 No eventing-con process running
2021/12/17 14:06:03 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/17 14:06:04 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:06:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:06:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:06:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:06:16 Starting bucket ops to source bucket
2021/12/17 14:06:17 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:06:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/17 14:06:23 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 14:06:28 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 14:06:33 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 14:06:38 Waiting for app: TestFlexReset1 to get deployed
2021/12/17 14:06:38 App: TestFlexReset1 got deployed
2021/12/17 14:06:43 Waiting for dst bucket item count to get to: 1000 curr count: 13
2021/12/17 14:06:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 14:06:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:06:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:06:48 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 14:06:53 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 14:06:58 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 14:07:03 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/17 14:07:03 App: TestFlexReset1 got un-deployed
2021/12/17 14:07:03 No eventing-con process running
2021/12/17 14:07:03 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/17 14:07:05 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:07:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:07:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:07:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:07:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:07:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.34s)
=== RUN   TestFlexReset2
2021/12/17 14:07:32 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 14:07:32 App: TestFlexReset2 got un-deployed
2021/12/17 14:07:32 No eventing-con process running
2021/12/17 14:07:32 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/17 14:07:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:07:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:07:40 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:07:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:07:45 Starting bucket ops to source bucket
2021/12/17 14:07:46 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:07:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/17 14:07:52 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 14:07:57 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 14:08:02 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 14:08:07 Waiting for app: TestFlexReset2 to get deployed
2021/12/17 14:08:07 App: TestFlexReset2 got deployed
2021/12/17 14:08:12 Waiting for dst bucket item count to get to: 1000 curr count: 74
2021/12/17 14:08:12 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/17 14:08:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:08:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:08:17 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 14:08:22 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 14:08:27 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 14:08:32 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/17 14:08:32 App: TestFlexReset2 got un-deployed
2021/12/17 14:08:32 No eventing-con process running
2021/12/17 14:08:33 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/17 14:08:34 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:08:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:08:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:08:46 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:08:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/17 14:08:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.25s)
=== RUN   TestN1QLLabelledBreak
2021/12/17 14:09:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 14:09:01 App: TestN1QLLabelledBreak got un-deployed
2021/12/17 14:09:01 No eventing-con process running
2021/12/17 14:09:01 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/17 14:09:03 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:09:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:09:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:09:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:09:15 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:09:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/17 14:09:21 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 14:09:26 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 14:09:31 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 14:09:36 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/17 14:09:36 App: TestN1QLLabelledBreak got deployed
2021/12/17 14:09:36 Starting bucket ops to source bucket
2021/12/17 14:09:45 Waiting for dst bucket item count to get to: 10000 curr count: 315
2021/12/17 14:09:50 Waiting for dst bucket item count to get to: 10000 curr count: 315
2021/12/17 14:09:55 Waiting for dst bucket item count to get to: 10000 curr count: 315
2021/12/17 14:10:00 Waiting for dst bucket item count to get to: 10000 curr count: 3759
2021/12/17 14:10:05 Waiting for dst bucket item count to get to: 10000 curr count: 3759
2021/12/17 14:10:10 Waiting for dst bucket item count to get to: 10000 curr count: 7803
2021/12/17 14:10:15 Waiting for dst bucket item count to get to: 10000 curr count: 7803
2021/12/17 14:10:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 14:10:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:10:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:10:21 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 14:10:26 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 14:10:31 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 14:10:36 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/17 14:10:36 App: TestN1QLLabelledBreak got un-deployed
2021/12/17 14:10:36 No eventing-con process running
2021/12/17 14:10:36 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/17 14:10:38 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:10:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/17 14:10:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:10:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:10:59 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 14:11:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:11:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.25s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/17 14:11:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 14:11:09 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/17 14:11:09 No eventing-con process running
2021/12/17 14:11:09 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/17 14:11:11 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:11:18 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:11:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:11:23 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:11:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/17 14:11:29 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 14:11:34 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 14:11:39 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 14:11:44 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/17 14:11:44 App: TestN1QLUnlabelledBreak got deployed
2021/12/17 14:11:44 Starting bucket ops to source bucket
2021/12/17 14:11:54 Waiting for dst bucket item count to get to: 10000 curr count: 324
2021/12/17 14:11:59 Waiting for dst bucket item count to get to: 10000 curr count: 957
2021/12/17 14:12:04 Waiting for dst bucket item count to get to: 10000 curr count: 957
2021/12/17 14:12:09 Waiting for dst bucket item count to get to: 10000 curr count: 3561
2021/12/17 14:12:14 Waiting for dst bucket item count to get to: 10000 curr count: 6458
2021/12/17 14:12:19 Waiting for dst bucket item count to get to: 10000 curr count: 8530
2021/12/17 14:12:19 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 14:12:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:12:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:12:24 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 14:12:29 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 14:12:34 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 14:12:39 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/17 14:12:39 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/17 14:12:39 No eventing-con process running
2021/12/17 14:12:39 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/17 14:12:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:12:51 Waiting for dst bucket item count to get to: 0 curr count: 4557
2021/12/17 14:12:56 Waiting for dst bucket item count to get to: 0 curr count: 4557
2021/12/17 14:12:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:12:58 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:13:08 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:13:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (123.27s)
=== RUN   TestN1QLThrowStatement
2021/12/17 14:13:13 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 14:13:13 App: TestN1QLThrowStatement got un-deployed
2021/12/17 14:13:13 No eventing-con process running
2021/12/17 14:13:13 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/17 14:13:14 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:13:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:13:26 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/17 14:13:32 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 14:13:37 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 14:13:42 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 14:13:47 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/17 14:13:47 App: TestN1QLThrowStatement got deployed
2021/12/17 14:13:47 Starting bucket ops to source bucket
2021/12/17 14:13:57 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/17 14:14:02 Waiting for dst bucket item count to get to: 10000 curr count: 126
2021/12/17 14:14:07 Waiting for dst bucket item count to get to: 10000 curr count: 126
2021/12/17 14:14:12 Waiting for dst bucket item count to get to: 10000 curr count: 3199
2021/12/17 14:14:17 Waiting for dst bucket item count to get to: 10000 curr count: 3199
2021/12/17 14:14:22 Waiting for dst bucket item count to get to: 10000 curr count: 7204
2021/12/17 14:14:27 Waiting for dst bucket item count to get to: 10000 curr count: 7204
2021/12/17 14:14:27 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/17 14:14:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:14:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:14:32 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 14:14:37 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 14:14:42 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 14:14:47 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/17 14:14:47 App: TestN1QLThrowStatement got un-deployed
2021/12/17 14:14:47 No eventing-con process running
2021/12/17 14:14:47 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/17 14:14:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:14:59 Waiting for dst bucket item count to get to: 0 curr count: 4515
2021/12/17 14:15:04 Waiting for dst bucket item count to get to: 0 curr count: 2015
2021/12/17 14:15:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:15:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:15:16 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/17 14:15:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (127.99s)
=== RUN   TestN1QLNestedForLoop
2021/12/17 14:15:21 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 14:15:21 App: TestN1QLNestedForLoop got un-deployed
2021/12/17 14:15:21 No eventing-con process running
2021/12/17 14:15:21 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/17 14:15:22 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:15:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:15:29 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:15:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:15:34 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:15:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/17 14:15:40 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 14:15:45 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 14:15:50 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 14:15:55 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/17 14:15:55 App: TestN1QLNestedForLoop got deployed
2021/12/17 14:15:55 Starting bucket ops to source bucket
2021/12/17 14:16:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/17 14:16:10 Waiting for dst bucket item count to get to: 5000 curr count: 198
2021/12/17 14:16:15 Waiting for dst bucket item count to get to: 5000 curr count: 198
2021/12/17 14:16:20 Waiting for dst bucket item count to get to: 5000 curr count: 2314
2021/12/17 14:16:25 Waiting for dst bucket item count to get to: 5000 curr count: 2314
2021/12/17 14:16:30 Waiting for dst bucket item count to get to: 5000 curr count: 4726
2021/12/17 14:16:35 Waiting for dst bucket item count to get to: 5000 curr count: 4726
2021/12/17 14:16:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 14:16:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:16:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:16:40 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 14:16:45 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 14:16:50 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 14:16:55 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/17 14:16:55 App: TestN1QLNestedForLoop got un-deployed
2021/12/17 14:16:55 No eventing-con process running
2021/12/17 14:16:55 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/17 14:16:57 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:17:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:17:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:17:09 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:17:19 Waiting for dst bucket item count to get to: 0 curr count: 4856
2021/12/17 14:17:24 Waiting for dst bucket item count to get to: 0 curr count: 2356
2021/12/17 14:17:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.27s)
=== RUN   TestN1QLAutoClose
2021/12/17 14:17:29 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 14:17:29 App: TestN1QLAutoClose got un-deployed
2021/12/17 14:17:29 No eventing-con process running
2021/12/17 14:17:29 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/17 14:17:31 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:17:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:17:42 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:17:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/17 14:17:49 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 14:17:54 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 14:17:59 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 14:18:04 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/17 14:18:04 App: TestN1QLAutoClose got deployed
2021/12/17 14:18:04 Starting bucket ops to source bucket
2021/12/17 14:18:09 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/17 14:18:14 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/17 14:18:19 Waiting for dst bucket item count to get to: 500 curr count: 9
2021/12/17 14:18:24 Waiting for dst bucket item count to get to: 500 curr count: 18
2021/12/17 14:18:29 Waiting for dst bucket item count to get to: 500 curr count: 31
2021/12/17 14:18:34 Waiting for dst bucket item count to get to: 500 curr count: 43
2021/12/17 14:18:39 Waiting for dst bucket item count to get to: 500 curr count: 54
2021/12/17 14:18:44 Waiting for dst bucket item count to get to: 500 curr count: 63
2021/12/17 14:18:49 Waiting for dst bucket item count to get to: 500 curr count: 75
2021/12/17 14:18:54 Waiting for dst bucket item count to get to: 500 curr count: 87
2021/12/17 14:18:59 Waiting for dst bucket item count to get to: 500 curr count: 97
2021/12/17 14:19:04 Waiting for dst bucket item count to get to: 500 curr count: 105
2021/12/17 14:19:09 Waiting for dst bucket item count to get to: 500 curr count: 105
2021/12/17 14:19:14 Waiting for dst bucket item count to get to: 500 curr count: 118
2021/12/17 14:19:19 Waiting for dst bucket item count to get to: 500 curr count: 127
2021/12/17 14:19:24 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/17 14:19:29 Waiting for dst bucket item count to get to: 500 curr count: 160
2021/12/17 14:19:34 Waiting for dst bucket item count to get to: 500 curr count: 160
2021/12/17 14:19:39 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/17 14:19:44 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/17 14:19:49 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/17 14:19:54 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/17 14:19:59 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/17 14:20:04 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/17 14:20:09 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/17 14:20:14 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/17 14:20:19 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/17 14:20:24 Waiting for dst bucket item count to get to: 500 curr count: 266
2021/12/17 14:20:29 Waiting for dst bucket item count to get to: 500 curr count: 288
2021/12/17 14:20:34 Waiting for dst bucket item count to get to: 500 curr count: 288
2021/12/17 14:20:39 Waiting for dst bucket item count to get to: 500 curr count: 310
2021/12/17 14:20:44 Waiting for dst bucket item count to get to: 500 curr count: 310
2021/12/17 14:20:49 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/17 14:20:54 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/17 14:20:59 Waiting for dst bucket item count to get to: 500 curr count: 353
2021/12/17 14:21:04 Waiting for dst bucket item count to get to: 500 curr count: 353
2021/12/17 14:21:09 Waiting for dst bucket item count to get to: 500 curr count: 372
2021/12/17 14:21:14 Waiting for dst bucket item count to get to: 500 curr count: 372
2021/12/17 14:21:19 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/17 14:21:24 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/17 14:21:29 Waiting for dst bucket item count to get to: 500 curr count: 413
2021/12/17 14:21:35 Waiting for dst bucket item count to get to: 500 curr count: 413
2021/12/17 14:21:40 Waiting for dst bucket item count to get to: 500 curr count: 434
2021/12/17 14:21:45 Waiting for dst bucket item count to get to: 500 curr count: 434
2021/12/17 14:21:50 Waiting for dst bucket item count to get to: 500 curr count: 455
2021/12/17 14:21:55 Waiting for dst bucket item count to get to: 500 curr count: 466
2021/12/17 14:22:00 Waiting for dst bucket item count to get to: 500 curr count: 479
2021/12/17 14:22:05 Waiting for dst bucket item count to get to: 500 curr count: 488
2021/12/17 14:22:10 Waiting for dst bucket item count to get to: 500 curr count: 497
2021/12/17 14:22:10 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/17 14:22:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:22:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:22:15 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 14:22:20 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 14:22:25 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 14:22:30 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/17 14:22:30 App: TestN1QLAutoClose got un-deployed
2021/12/17 14:22:30 No eventing-con process running
2021/12/17 14:22:30 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/17 14:22:32 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:22:42 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 14:22:47 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/17 14:22:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:22:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:22:58 Waiting for dst bucket item count to get to: 0 curr count: 403
2021/12/17 14:23:03 Waiting for dst bucket item count to get to: 0 curr count: 153
2021/12/17 14:23:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.52s)
=== RUN   TestN1QLPosParams
2021/12/17 14:23:08 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 14:23:08 App: TestN1QLPosParams got un-deployed
2021/12/17 14:23:08 No eventing-con process running
2021/12/17 14:23:08 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/17 14:23:10 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:23:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:23:17 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:23:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:23:22 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:23:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/17 14:23:28 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 14:23:33 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 14:23:38 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 14:23:43 Waiting for app: TestN1QLPosParams to get deployed
2021/12/17 14:23:43 App: TestN1QLPosParams got deployed
2021/12/17 14:23:43 Starting bucket ops to source bucket
2021/12/17 14:23:53 Waiting for dst bucket item count to get to: 5000 curr count: 669
2021/12/17 14:23:58 Waiting for dst bucket item count to get to: 5000 curr count: 2010
2021/12/17 14:24:03 Waiting for dst bucket item count to get to: 5000 curr count: 3841
2021/12/17 14:24:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 14:24:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:24:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:24:08 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 14:24:13 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 14:24:18 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 14:24:23 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/17 14:24:23 App: TestN1QLPosParams got un-deployed
2021/12/17 14:24:23 No eventing-con process running
2021/12/17 14:24:23 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/17 14:24:25 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:24:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:24:37 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:24:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (103.26s)
=== RUN   TestN1QLExhaustConnPool
2021/12/17 14:24:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 14:24:52 App: TestN1QLExhaustConnPool got un-deployed
2021/12/17 14:24:52 No eventing-con process running
2021/12/17 14:24:52 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/17 14:24:53 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:25:00 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:25:05 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:25:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/17 14:25:11 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 14:25:16 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 14:25:21 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 14:25:26 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/17 14:25:26 App: TestN1QLExhaustConnPool got deployed
2021/12/17 14:25:26 Starting bucket ops to source bucket
2021/12/17 14:25:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/17 14:25:37 Waiting for dst bucket item count to get to: 100 curr count: 51
2021/12/17 14:25:42 Waiting for dst bucket item count to get to: 100 curr count: 51
2021/12/17 14:25:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/17 14:25:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:25:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:25:47 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 14:25:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 14:25:57 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 14:26:02 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/17 14:26:02 App: TestN1QLExhaustConnPool got un-deployed
2021/12/17 14:26:02 No eventing-con process running
2021/12/17 14:26:02 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/17 14:26:04 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:26:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/17 14:26:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/17 14:26:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:26:21 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:26:31 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/17 14:26:36 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/17 14:26:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.89s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/17 14:26:41 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 14:26:41 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/17 14:26:41 No eventing-con process running
2021/12/17 14:26:41 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/17 14:26:42 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:26:49 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:26:54 Starting bucket ops to source bucket
2021/12/17 14:27:12 Starting bucket ops to source bucket
2021/12/17 14:27:29 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 11204
2021/12/17 14:27:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/17 14:27:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 14:27:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 14:27:45 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 14:27:50 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/17 14:27:50 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/17 14:27:50 Starting bucket ops to source bucket
2021/12/17 14:27:55 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:00 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:05 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:10 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:15 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:20 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:26 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:31 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:36 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:41 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/17 14:28:46 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/17 14:28:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:28:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:28:51 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 14:28:56 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 14:29:01 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 14:29:06 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/17 14:29:06 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/17 14:29:06 No eventing-con process running
2021/12/17 14:29:06 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/17 14:29:08 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:29:18 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/17 14:29:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:29:19 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:29:29 Waiting for dst bucket item count to get to: 0 curr count: 15001
2021/12/17 14:29:34 Waiting for dst bucket item count to get to: 0 curr count: 15001
2021/12/17 14:29:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (178.89s)
=== RUN   TestN1QLQueryConsistency
2021/12/17 14:29:39 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 14:29:39 App: n1qlquery_consistency got un-deployed
2021/12/17 14:29:39 No eventing-con process running
2021/12/17 14:29:39 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/17 14:29:41 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:29:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:29:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:29:53 Starting bucket ops to source bucket
2021/12/17 14:29:56 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:29:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/17 14:30:02 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 14:30:07 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 14:30:12 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 14:30:17 Waiting for app: n1qlquery_consistency to get deployed
2021/12/17 14:30:17 App: n1qlquery_consistency got deployed
2021/12/17 14:30:22 Waiting for dst bucket item count to get to: 5000 curr count: 52
2021/12/17 14:30:27 Waiting for dst bucket item count to get to: 5000 curr count: 2477
2021/12/17 14:30:32 Waiting for dst bucket item count to get to: 5000 curr count: 4925
2021/12/17 14:30:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/17 14:30:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/17 14:30:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:30:37 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 14:30:42 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 14:30:47 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 14:30:52 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/17 14:30:52 App: n1qlquery_consistency got un-deployed
2021/12/17 14:30:52 No eventing-con process running
2021/12/17 14:30:53 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/17 14:30:54 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:31:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:31:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:31:06 Context: Bucket flush, response code: 200 dump: 
2021/12/17 14:31:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/17 14:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.71s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1549.170s

N1ql tests took 0 hours, 25 minutes and 53 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/17 14:32:52 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:50306->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:50306->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/17 14:33:10 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/17 14:33:15 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 14:33:15 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/17 14:33:15 No eventing-con process running
2021/12/17 14:33:15 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/17 14:33:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:33:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:33:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:33:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:33:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:33:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:33:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:33:34 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:33:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 14:33:40 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 14:33:45 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 14:33:50 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 14:33:55 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/17 14:33:55 App: TestDuplicateWithEventingRebIn got deployed
2021/12/17 14:33:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 14:33:55 Starting bucket ops to source bucket
2021/12/17 14:34:10 Removing node(s): [] from the cluster
2021/12/17 14:34:15 Rebalance progress: 0
2021/12/17 14:34:20 Rebalance progress: 0
2021/12/17 14:34:25 Rebalance progress: 25
2021/12/17 14:34:30 Rebalance progress: 25
2021/12/17 14:34:35 Rebalance progress: 25
2021/12/17 14:34:40 Rebalance progress: 98.52941176470588
2021/12/17 14:34:45 Rebalance progress: 98.52941176470588
2021/12/17 14:34:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:34:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:34:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:34:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 14:34:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:34:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:34:55 Rebalance progress: 25
2021/12/17 14:35:00 Rebalance progress: 25
2021/12/17 14:35:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:35:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:35:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:35:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 14:35:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:35:05 Waiting for src bucket item count to get to: 200000 curr count: 104660
2021/12/17 14:35:10 Waiting for src bucket item count to get to: 200000 curr count: 104660
2021/12/17 14:35:15 Waiting for src bucket item count to get to: 200000 curr count: 121245
2021/12/17 14:35:20 Waiting for src bucket item count to get to: 200000 curr count: 121245
2021/12/17 14:35:25 Waiting for src bucket item count to get to: 200000 curr count: 137814
2021/12/17 14:35:30 Waiting for src bucket item count to get to: 200000 curr count: 137814
2021/12/17 14:35:35 Waiting for src bucket item count to get to: 200000 curr count: 154488
2021/12/17 14:35:40 Waiting for src bucket item count to get to: 200000 curr count: 154488
2021/12/17 14:35:45 Waiting for src bucket item count to get to: 200000 curr count: 170652
2021/12/17 14:35:50 Waiting for src bucket item count to get to: 200000 curr count: 170652
2021/12/17 14:35:55 Waiting for src bucket item count to get to: 200000 curr count: 187040
2021/12/17 14:36:00 Waiting for src bucket item count to get to: 200000 curr count: 187040
2021/12/17 14:36:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 14:36:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 14:36:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:36:10 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 14:36:15 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 14:36:20 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 14:36:25 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/17 14:36:25 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/17 14:36:25 No eventing-con process running
2021/12/17 14:36:25 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/17 14:36:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:36:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:36:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:36:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:36:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:36:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:36:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:36:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (228.27s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/17 14:36:53 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 14:36:54 Removing node(s): [] from the cluster
2021/12/17 14:36:59 Rebalance progress: 100
2021/12/17 14:37:04 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 14:37:04 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/17 14:37:04 No eventing-con process running
2021/12/17 14:37:04 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/17 14:37:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:37:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:37:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:37:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:37:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:37:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:37:22 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:37:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 14:37:28 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 14:37:33 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 14:37:38 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 14:37:43 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/17 14:37:43 App: TestDuplicateWithEventingRebOut got deployed
2021/12/17 14:37:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:37:43 Starting bucket ops to source bucket
2021/12/17 14:37:48 Rebalance progress: 25
2021/12/17 14:37:53 Rebalance progress: 25
2021/12/17 14:37:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:37:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:37:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 14:37:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:37:58 Waiting for src bucket item count to get to: 200000 curr count: 9311
2021/12/17 14:38:03 Waiting for src bucket item count to get to: 200000 curr count: 25627
2021/12/17 14:38:08 Waiting for src bucket item count to get to: 200000 curr count: 25627
2021/12/17 14:38:13 Waiting for src bucket item count to get to: 200000 curr count: 41817
2021/12/17 14:38:18 Waiting for src bucket item count to get to: 200000 curr count: 41817
2021/12/17 14:38:23 Waiting for src bucket item count to get to: 200000 curr count: 58307
2021/12/17 14:38:28 Waiting for src bucket item count to get to: 200000 curr count: 58307
2021/12/17 14:38:33 Waiting for src bucket item count to get to: 200000 curr count: 58307
2021/12/17 14:38:38 Waiting for src bucket item count to get to: 200000 curr count: 74689
2021/12/17 14:38:43 Waiting for src bucket item count to get to: 200000 curr count: 74689
2021/12/17 14:38:48 Waiting for src bucket item count to get to: 200000 curr count: 90829
2021/12/17 14:38:54 Waiting for src bucket item count to get to: 200000 curr count: 90829
2021/12/17 14:38:59 Waiting for src bucket item count to get to: 200000 curr count: 107264
2021/12/17 14:39:04 Waiting for src bucket item count to get to: 200000 curr count: 107264
2021/12/17 14:39:09 Waiting for src bucket item count to get to: 200000 curr count: 123709
2021/12/17 14:39:14 Waiting for src bucket item count to get to: 200000 curr count: 123709
2021/12/17 14:39:19 Waiting for src bucket item count to get to: 200000 curr count: 140157
2021/12/17 14:39:24 Waiting for src bucket item count to get to: 200000 curr count: 140157
2021/12/17 14:39:29 Waiting for src bucket item count to get to: 200000 curr count: 156332
2021/12/17 14:39:34 Waiting for src bucket item count to get to: 200000 curr count: 156332
2021/12/17 14:39:39 Waiting for src bucket item count to get to: 200000 curr count: 172311
2021/12/17 14:39:44 Waiting for src bucket item count to get to: 200000 curr count: 188381
2021/12/17 14:39:49 Waiting for src bucket item count to get to: 200000 curr count: 188381
2021/12/17 14:39:54 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 14:39:54 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 14:39:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:39:59 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 14:40:04 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 14:40:09 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 14:40:14 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/17 14:40:14 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/17 14:40:14 No eventing-con process running
2021/12/17 14:40:14 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/17 14:40:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:40:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:40:25 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:40:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:40:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:40:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:40:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.79s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/17 14:40:47 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 14:40:47 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/17 14:40:47 No eventing-con process running
2021/12/17 14:40:47 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/17 14:40:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:40:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:40:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:40:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:41:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:41:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:41:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:41:06 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:41:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 14:41:12 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 14:41:17 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 14:41:22 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 14:41:27 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/17 14:41:27 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/17 14:41:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 14:41:27 Starting bucket ops to source bucket
2021/12/17 14:41:27 Removing node(s): [] from the cluster
2021/12/17 14:41:32 Rebalance progress: 25
2021/12/17 14:41:37 Rebalance progress: 25
2021/12/17 14:41:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:41:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/17 14:41:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:41:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 14:41:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:41:55 Rebalance progress: 16.66666666666667
2021/12/17 14:42:00 Rebalance progress: 16.66666666666667
2021/12/17 14:42:05 Rebalance progress: 16.66666666666667
2021/12/17 14:42:10 Rebalance progress: 29.43262411347517
2021/12/17 14:42:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:42:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:42:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:42:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:42:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 14:42:20 Rebalance progress: 16.66666666666667
2021/12/17 14:42:25 Rebalance progress: 16.66666666666667
2021/12/17 14:42:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:42:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:42:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:42:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:42:30 Waiting for src bucket item count to get to: 200000 curr count: 78323
2021/12/17 14:42:35 Waiting for src bucket item count to get to: 200000 curr count: 94191
2021/12/17 14:42:40 Waiting for src bucket item count to get to: 200000 curr count: 94191
2021/12/17 14:42:45 Waiting for src bucket item count to get to: 200000 curr count: 110590
2021/12/17 14:42:50 Waiting for src bucket item count to get to: 200000 curr count: 110590
2021/12/17 14:42:55 Waiting for src bucket item count to get to: 200000 curr count: 127138
2021/12/17 14:43:00 Waiting for src bucket item count to get to: 200000 curr count: 127138
2021/12/17 14:43:05 Waiting for src bucket item count to get to: 200000 curr count: 143437
2021/12/17 14:43:10 Waiting for src bucket item count to get to: 200000 curr count: 143437
2021/12/17 14:43:15 Waiting for src bucket item count to get to: 200000 curr count: 159487
2021/12/17 14:43:20 Waiting for src bucket item count to get to: 200000 curr count: 159487
2021/12/17 14:43:25 Waiting for src bucket item count to get to: 200000 curr count: 175131
2021/12/17 14:43:30 Waiting for src bucket item count to get to: 200000 curr count: 175131
2021/12/17 14:43:35 Waiting for src bucket item count to get to: 200000 curr count: 191081
2021/12/17 14:43:40 Waiting for src bucket item count to get to: 200000 curr count: 191081
2021/12/17 14:43:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 14:43:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 14:43:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:43:50 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 14:43:55 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 14:44:00 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 14:44:05 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/17 14:44:05 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/17 14:44:05 No eventing-con process running
2021/12/17 14:44:05 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/17 14:44:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:44:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:44:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:44:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:44:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:44:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.26s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/17 14:44:38 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 14:44:38 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/17 14:44:39 No eventing-con process running
2021/12/17 14:44:39 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/17 14:44:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:44:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:44:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:44:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:44:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:44:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:44:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:44:57 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:44:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 14:45:03 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 14:45:08 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 14:45:13 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 14:45:18 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/17 14:45:18 App: TestDuplicateWithKVRebIn got deployed
2021/12/17 14:45:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 14:45:18 Starting bucket ops to source bucket
2021/12/17 14:45:18 Removing node(s): [] from the cluster
2021/12/17 14:45:23 Rebalance progress: 16.66666666666667
2021/12/17 14:45:28 Rebalance progress: 16.66666666666667
2021/12/17 14:45:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:45:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:45:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:45:39 Rebalance progress: 16.66666666666667
2021/12/17 14:45:44 Rebalance progress: 16.66666666666667
2021/12/17 14:45:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:45:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:45:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:45:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:45:49 Waiting for src bucket item count to get to: 200000 curr count: 32497
2021/12/17 14:45:54 Waiting for src bucket item count to get to: 200000 curr count: 48897
2021/12/17 14:45:59 Waiting for src bucket item count to get to: 200000 curr count: 48897
2021/12/17 14:46:04 Waiting for src bucket item count to get to: 200000 curr count: 65225
2021/12/17 14:46:09 Waiting for src bucket item count to get to: 200000 curr count: 65225
2021/12/17 14:46:14 Waiting for src bucket item count to get to: 200000 curr count: 80827
2021/12/17 14:46:19 Waiting for src bucket item count to get to: 200000 curr count: 80827
2021/12/17 14:46:24 Waiting for src bucket item count to get to: 200000 curr count: 80827
2021/12/17 14:46:29 Waiting for src bucket item count to get to: 200000 curr count: 96969
2021/12/17 14:46:34 Waiting for src bucket item count to get to: 200000 curr count: 96969
2021/12/17 14:46:39 Waiting for src bucket item count to get to: 200000 curr count: 113310
2021/12/17 14:46:44 Waiting for src bucket item count to get to: 200000 curr count: 113310
2021/12/17 14:46:49 Waiting for src bucket item count to get to: 200000 curr count: 129596
2021/12/17 14:46:54 Waiting for src bucket item count to get to: 200000 curr count: 129596
2021/12/17 14:46:59 Waiting for src bucket item count to get to: 200000 curr count: 145790
2021/12/17 14:47:04 Waiting for src bucket item count to get to: 200000 curr count: 145790
2021/12/17 14:47:09 Waiting for src bucket item count to get to: 200000 curr count: 161809
2021/12/17 14:47:14 Waiting for src bucket item count to get to: 200000 curr count: 161809
2021/12/17 14:47:19 Waiting for src bucket item count to get to: 200000 curr count: 177074
2021/12/17 14:47:24 Waiting for src bucket item count to get to: 200000 curr count: 177074
2021/12/17 14:47:29 Waiting for src bucket item count to get to: 200000 curr count: 192779
2021/12/17 14:47:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 14:47:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 14:47:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:47:39 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 14:47:44 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 14:47:49 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 14:47:54 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/17 14:47:54 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/17 14:47:54 No eventing-con process running
2021/12/17 14:47:54 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/17 14:47:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:48:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:48:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:48:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:48:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:48:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:48:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (228.90s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/17 14:48:22 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 14:48:23 Removing node(s): [] from the cluster
2021/12/17 14:48:28 Rebalance progress: 100
2021/12/17 14:48:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 14:48:33 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/17 14:48:33 No eventing-con process running
2021/12/17 14:48:33 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/17 14:48:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:48:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:48:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:48:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:48:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:48:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:48:51 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:48:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 14:48:57 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 14:49:02 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 14:49:07 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 14:49:12 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/17 14:49:12 App: TestDuplicateWithKVRebOut got deployed
2021/12/17 14:49:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:49:12 Starting bucket ops to source bucket
2021/12/17 14:49:18 Rebalance progress: 16.66666666666667
2021/12/17 14:49:23 Rebalance progress: 16.66666666666667
2021/12/17 14:49:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:49:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:49:28 Waiting for src bucket item count to get to: 200000 curr count: 10207
2021/12/17 14:49:33 Waiting for src bucket item count to get to: 200000 curr count: 26483
2021/12/17 14:49:38 Waiting for src bucket item count to get to: 200000 curr count: 26483
2021/12/17 14:49:43 Waiting for src bucket item count to get to: 200000 curr count: 42880
2021/12/17 14:49:48 Waiting for src bucket item count to get to: 200000 curr count: 42880
2021/12/17 14:49:53 Waiting for src bucket item count to get to: 200000 curr count: 59101
2021/12/17 14:49:58 Waiting for src bucket item count to get to: 200000 curr count: 59101
2021/12/17 14:50:03 Waiting for src bucket item count to get to: 200000 curr count: 59101
2021/12/17 14:50:08 Waiting for src bucket item count to get to: 200000 curr count: 74624
2021/12/17 14:50:13 Waiting for src bucket item count to get to: 200000 curr count: 74624
2021/12/17 14:50:18 Waiting for src bucket item count to get to: 200000 curr count: 90475
2021/12/17 14:50:23 Waiting for src bucket item count to get to: 200000 curr count: 90475
2021/12/17 14:50:28 Waiting for src bucket item count to get to: 200000 curr count: 106750
2021/12/17 14:50:33 Waiting for src bucket item count to get to: 200000 curr count: 106750
2021/12/17 14:50:38 Waiting for src bucket item count to get to: 200000 curr count: 123007
2021/12/17 14:50:43 Waiting for src bucket item count to get to: 200000 curr count: 123007
2021/12/17 14:50:48 Waiting for src bucket item count to get to: 200000 curr count: 139412
2021/12/17 14:50:53 Waiting for src bucket item count to get to: 200000 curr count: 139412
2021/12/17 14:50:58 Waiting for src bucket item count to get to: 200000 curr count: 155820
2021/12/17 14:51:03 Waiting for src bucket item count to get to: 200000 curr count: 155820
2021/12/17 14:51:08 Waiting for src bucket item count to get to: 200000 curr count: 171617
2021/12/17 14:51:13 Waiting for src bucket item count to get to: 200000 curr count: 171617
2021/12/17 14:51:18 Waiting for src bucket item count to get to: 200000 curr count: 187473
2021/12/17 14:51:23 Waiting for src bucket item count to get to: 200000 curr count: 187473
2021/12/17 14:51:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 14:51:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 14:51:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:51:33 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 14:51:38 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 14:51:43 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 14:51:48 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/17 14:51:48 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/17 14:51:48 No eventing-con process running
2021/12/17 14:51:48 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/17 14:51:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:51:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:51:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:52:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:52:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.02s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/17 14:52:16 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 14:52:16 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/17 14:52:16 No eventing-con process running
2021/12/17 14:52:16 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/17 14:52:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:52:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:52:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:52:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:52:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:52:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:52:35 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:52:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 14:52:41 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 14:52:46 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 14:52:51 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 14:52:56 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/17 14:52:56 App: TestDuplicateWithKVSwapReb got deployed
2021/12/17 14:52:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 14:52:56 Starting bucket ops to source bucket
2021/12/17 14:52:56 Removing node(s): [] from the cluster
2021/12/17 14:53:01 Rebalance progress: 16.66666666666667
2021/12/17 14:53:06 Rebalance progress: 16.66666666666667
2021/12/17 14:53:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:53:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:53:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:53:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:53:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 14:53:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:53:17 Rebalance progress: 16.66666666666667
2021/12/17 14:53:22 Rebalance progress: 16.66666666666667
2021/12/17 14:53:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:53:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:53:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:53:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:53:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 14:53:32 Rebalance progress: 16.66666666666667
2021/12/17 14:53:37 Rebalance progress: 16.66666666666667
2021/12/17 14:53:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:53:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:53:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:53:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:53:42 Waiting for src bucket item count to get to: 200000 curr count: 67507
2021/12/17 14:53:47 Waiting for src bucket item count to get to: 200000 curr count: 67507
2021/12/17 14:53:52 Waiting for src bucket item count to get to: 200000 curr count: 82570
2021/12/17 14:53:57 Waiting for src bucket item count to get to: 200000 curr count: 82570
2021/12/17 14:54:02 Waiting for src bucket item count to get to: 200000 curr count: 98829
2021/12/17 14:54:07 Waiting for src bucket item count to get to: 200000 curr count: 98829
2021/12/17 14:54:12 Waiting for src bucket item count to get to: 200000 curr count: 114915
2021/12/17 14:54:17 Waiting for src bucket item count to get to: 200000 curr count: 114915
2021/12/17 14:54:22 Waiting for src bucket item count to get to: 200000 curr count: 114915
2021/12/17 14:54:27 Waiting for src bucket item count to get to: 200000 curr count: 131089
2021/12/17 14:54:32 Waiting for src bucket item count to get to: 200000 curr count: 131089
2021/12/17 14:54:37 Waiting for src bucket item count to get to: 200000 curr count: 147605
2021/12/17 14:54:42 Waiting for src bucket item count to get to: 200000 curr count: 147605
2021/12/17 14:54:47 Waiting for src bucket item count to get to: 200000 curr count: 163948
2021/12/17 14:54:52 Waiting for src bucket item count to get to: 200000 curr count: 163948
2021/12/17 14:54:57 Waiting for src bucket item count to get to: 200000 curr count: 179309
2021/12/17 14:55:02 Waiting for src bucket item count to get to: 200000 curr count: 179309
2021/12/17 14:55:07 Waiting for src bucket item count to get to: 200000 curr count: 195758
2021/12/17 14:55:12 Waiting for src bucket item count to get to: 200000 curr count: 195758
2021/12/17 14:55:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 14:55:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 14:55:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:55:22 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 14:55:27 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 14:55:32 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 14:55:37 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/17 14:55:37 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/17 14:55:37 No eventing-con process running
2021/12/17 14:55:37 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/17 14:55:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:55:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:55:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/17 14:55:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:55:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:55:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:56:01 Waiting for dst bucket item count to get to: 0 curr count: 151287
2021/12/17 14:56:06 Waiting for dst bucket item count to get to: 0 curr count: 151287
2021/12/17 14:56:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.32s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/17 14:56:16 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 14:56:16 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 14:56:16 No eventing-con process running
2021/12/17 14:56:16 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/17 14:56:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:56:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:56:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:56:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:56:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:56:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:56:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:56:34 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:56:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/17 14:56:40 Starting bucket ops to source bucket
2021/12/17 14:56:45 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 14:56:50 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 14:56:55 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 14:56:55 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:56:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:56:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:56:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:56:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 14:56:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 14:57:01 Rebalance progress: 16.66666666666667
2021/12/17 14:57:06 Rebalance progress: 16.66666666666667
2021/12/17 14:57:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:57:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:57:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 14:57:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 14:57:17 Rebalance progress: 16.66666666666667
2021/12/17 14:57:22 Rebalance progress: 16.66666666666667
2021/12/17 14:57:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:57:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:57:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/17 14:57:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 14:57:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 14:57:37 Rebalance progress: 12.5
2021/12/17 14:57:42 Rebalance progress: 12.5
2021/12/17 14:57:47 Rebalance progress: 12.5
2021/12/17 14:57:52 Rebalance progress: 62.70491803278689
2021/12/17 14:57:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:57:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 14:58:02 Rebalance progress: 12.5
2021/12/17 14:58:07 Rebalance progress: 12.5
2021/12/17 14:58:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:58:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 14:58:17 Rebalance progress: 12.5
2021/12/17 14:58:22 Rebalance progress: 12.5
2021/12/17 14:58:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:58:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:58:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:58:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 14:58:32 Rebalance progress: 12.5
2021/12/17 14:58:37 Rebalance progress: 12.5
2021/12/17 14:58:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:58:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:58:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 14:58:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 14:58:48 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 14:58:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 14:58:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 14:59:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 14:59:03 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 14:59:03 eventingSetup.go:46: No eventing-con process running
2021/12/17 14:59:03 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/17 14:59:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:59:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:59:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12200
2021/12/17 14:59:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:59:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:59:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:59:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12200
2021/12/17 14:59:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.58s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/17 14:59:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 14:59:36 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/17 14:59:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 14:59:36 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/17 14:59:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:59:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:59:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 14:59:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:59:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:59:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 14:59:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 14:59:55 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/17 14:59:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/17 15:00:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:00:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/17 15:00:11 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/17 15:00:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/17 15:00:16 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:00:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:00:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:00:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:00:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:00:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 15:00:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:00:22 Rebalance progress: 12.5
2021/12/17 15:00:27 Rebalance progress: 12.5
2021/12/17 15:00:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:00:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 15:00:37 Rebalance progress: 12.5
2021/12/17 15:00:42 Rebalance progress: 12.5
2021/12/17 15:00:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:00:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:00:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 15:00:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 15:01:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 15:01:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/17 15:01:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/17 15:01:08 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:01:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/17 15:01:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:01:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:01:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:01:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:01:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:01:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4622
2021/12/17 15:01:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.88s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/17 15:01:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 15:01:36 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 15:01:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:01:36 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/17 15:01:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:01:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:01:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:01:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:01:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:01:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:01:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:01:55 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:01:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/17 15:02:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:02:06 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/17 15:02:11 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/17 15:02:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/17 15:02:16 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:02:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:02:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:02:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:02:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:02:21 Rebalance progress: 12.5
2021/12/17 15:02:26 Rebalance progress: 12.5
2021/12/17 15:02:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:02:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:02:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:02:37 Rebalance progress: 12.5
2021/12/17 15:02:42 Rebalance progress: 12.5
2021/12/17 15:02:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:02:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 15:02:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:02:52 Rebalance progress: 12.5
2021/12/17 15:02:57 Rebalance progress: 12.5
2021/12/17 15:03:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:03:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:03:07 Rebalance progress: 12.5
2021/12/17 15:03:12 Rebalance progress: 12.5
2021/12/17 15:03:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:03:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:03:22 Rebalance progress: 12.5
2021/12/17 15:03:27 Rebalance progress: 12.5
2021/12/17 15:03:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:03:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 15:03:37 Rebalance progress: 12.5
2021/12/17 15:03:42 Rebalance progress: 12.5
2021/12/17 15:03:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:03:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:03:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:03:53 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 15:03:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 15:04:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 15:04:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/17 15:04:08 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/17 15:04:08 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:04:08 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/17 15:04:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:04:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:04:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:04:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:04:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:04:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10632
2021/12/17 15:04:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (180.00s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/17 15:04:36 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 15:04:36 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/17 15:04:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:04:36 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/17 15:04:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:04:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:04:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:04:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:04:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:04:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:04:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:04:50 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:04:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 15:04:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:05:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/17 15:05:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/17 15:05:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/17 15:05:11 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:05:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:05:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:05:17 Rebalance progress: 12.5
2021/12/17 15:05:22 Rebalance progress: 12.5
2021/12/17 15:05:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:05:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:05:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:05:32 Rebalance progress: 12.5
2021/12/17 15:05:37 Rebalance progress: 12.5
2021/12/17 15:05:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:05:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:05:47 Rebalance progress: 12.5
2021/12/17 15:05:52 Rebalance progress: 12.5
2021/12/17 15:05:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:05:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:05:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:06:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 15:06:07 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 15:06:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 15:06:17 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/17 15:06:17 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/17 15:06:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:06:17 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/17 15:06:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:06:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:06:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6115
2021/12/17 15:06:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:06:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:06:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:06:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5436
2021/12/17 15:06:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5436
2021/12/17 15:06:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.69s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/17 15:06:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:06:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:06:57 Rebalance progress: 100
2021/12/17 15:07:02 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 15:07:02 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/17 15:07:02 No eventing-con process running
2021/12/17 15:07:02 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/17 15:07:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:07:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:07:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:07:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:07:15 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:07:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/17 15:07:21 Starting bucket ops to source bucket
2021/12/17 15:07:26 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/17 15:07:31 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/17 15:07:36 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/17 15:07:36 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:07:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:07:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:07:42 Rebalance progress: 12.5
2021/12/17 15:07:47 Rebalance progress: 12.5
2021/12/17 15:07:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:07:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:07:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 15:07:52 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/17 15:07:57 Rebalance progress: 12.5
2021/12/17 15:08:02 Rebalance progress: 12.5
2021/12/17 15:08:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:08:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:08:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:08:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:08:12 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 15:08:17 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 15:08:22 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 15:08:27 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/17 15:08:27 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/17 15:08:27 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:08:27 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/17 15:08:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:08:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:08:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4565
2021/12/17 15:08:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:08:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:08:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:08:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3834
2021/12/17 15:08:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3834
2021/12/17 15:08:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.95s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/17 15:09:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:09:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:09:06 Rebalance progress: 100
2021/12/17 15:09:11 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 15:09:11 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/17 15:09:11 No eventing-con process running
2021/12/17 15:09:11 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/17 15:09:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:09:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:09:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:09:25 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:09:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/17 15:09:31 Starting bucket ops to source bucket
2021/12/17 15:09:36 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/17 15:09:41 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/17 15:09:46 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/17 15:09:46 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:09:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:09:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:09:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:09:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:09:51 Rebalance progress: 12.5
2021/12/17 15:09:56 Rebalance progress: 12.5
2021/12/17 15:10:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:10:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 15:10:01 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/17 15:10:06 Rebalance progress: 12.5
2021/12/17 15:10:11 Rebalance progress: 12.5
2021/12/17 15:10:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:10:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:10:17 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/17 15:10:22 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 15:10:27 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 15:10:32 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 15:10:37 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/17 15:10:37 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/17 15:10:37 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:10:37 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/17 15:10:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:10:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:10:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37879
2021/12/17 15:10:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:10:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:10:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:11:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12903
2021/12/17 15:11:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12903
2021/12/17 15:11:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.43s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/17 15:11:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 15:11:15 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/17 15:11:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:11:15 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/17 15:11:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:11:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:11:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:11:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:11:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:11:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:11:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:11:34 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:11:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:11:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:11:35 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:11:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 15:11:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:11:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/17 15:11:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/17 15:11:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/17 15:11:56 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/17 15:12:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/17 15:12:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/17 15:12:11 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/17 15:12:11 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:12:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:12:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:12:17 Rebalance progress: 12.5
2021/12/17 15:12:22 Rebalance progress: 12.5
2021/12/17 15:12:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:12:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:12:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:12:32 Rebalance progress: 12.5
2021/12/17 15:12:37 Rebalance progress: 12.5
2021/12/17 15:12:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:12:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 15:12:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:12:47 Rebalance progress: 12.5
2021/12/17 15:12:52 Rebalance progress: 12.5
2021/12/17 15:12:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:12:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:12:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:13:02 Rebalance progress: 12.5
2021/12/17 15:13:07 Rebalance progress: 12.5
2021/12/17 15:13:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:13:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:13:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:13:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:13:18 Rebalance progress: 12.5
2021/12/17 15:13:23 Rebalance progress: 12.5
2021/12/17 15:13:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:13:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:13:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 15:13:33 Rebalance progress: 12.5
2021/12/17 15:13:38 Rebalance progress: 12.5
2021/12/17 15:13:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:13:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:13:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:13:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:13:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 15:13:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 15:13:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 15:14:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/17 15:14:03 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/17 15:14:03 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:14:03 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/17 15:14:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:14:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:14:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12147
2021/12/17 15:14:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:14:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:14:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:14:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12147
2021/12/17 15:14:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:14:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:14:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 15:14:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 15:14:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 15:14:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/17 15:14:48 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/17 15:14:48 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:14:48 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/17 15:14:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:14:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:14:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:14:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:15:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:15:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (236.60s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/17 15:15:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 15:15:12 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/17 15:15:12 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:15:12 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/17 15:15:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:15:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:15:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:15:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:15:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:15:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:15:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:15:31 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:15:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:15:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:15:32 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:15:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 15:15:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:15:43 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/17 15:15:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/17 15:15:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/17 15:15:53 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/17 15:15:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/17 15:16:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/17 15:16:08 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/17 15:16:08 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:16:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:16:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:16:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:16:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:16:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/17 15:16:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:16:14 Rebalance progress: 12.5
2021/12/17 15:16:19 Rebalance progress: 12.5
2021/12/17 15:16:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:16:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 15:16:29 Rebalance progress: 12.5
2021/12/17 15:16:34 Rebalance progress: 12.5
2021/12/17 15:16:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:16:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:16:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 15:16:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 15:16:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 15:17:00 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/17 15:17:00 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/17 15:17:00 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:17:00 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/17 15:17:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:17:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:17:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6134
2021/12/17 15:17:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6134
2021/12/17 15:17:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:17:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:17:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:17:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:17:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:17:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 15:17:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 15:17:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 15:17:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/17 15:17:45 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/17 15:17:45 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:17:45 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/17 15:17:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:17:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:17:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:17:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:17:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:17:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (176.71s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/17 15:18:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 15:18:09 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/17 15:18:09 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:18:09 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/17 15:18:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:18:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:18:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:18:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:18:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:18:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:18:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 15:18:27 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/17 15:18:27 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:18:27 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/17 15:18:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:18:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:18:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:18:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:18:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:18:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:18:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:18:41 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:18:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:18:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:18:42 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:18:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 15:18:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:18:53 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/17 15:18:58 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/17 15:19:03 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/17 15:19:03 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/17 15:19:08 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/17 15:19:13 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/17 15:19:18 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/17 15:19:18 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:19:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:19:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:19:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:19:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:19:24 Rebalance progress: 12.5
2021/12/17 15:19:29 Rebalance progress: 12.5
2021/12/17 15:19:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:19:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:19:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:19:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:19:39 Rebalance progress: 12.5
2021/12/17 15:19:44 Rebalance progress: 12.5
2021/12/17 15:19:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:19:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:19:54 Rebalance progress: 12.5
2021/12/17 15:19:59 Rebalance progress: 12.5
2021/12/17 15:20:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:20:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:20:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 15:20:14 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 15:20:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 15:20:24 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/17 15:20:24 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/17 15:20:24 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:20:24 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/17 15:20:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:20:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:20:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7575
2021/12/17 15:20:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:20:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:20:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:20:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1706
2021/12/17 15:20:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1706
2021/12/17 15:20:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:20:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:21:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 15:21:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 15:21:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 15:21:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/17 15:21:15 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/17 15:21:15 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:21:15 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/17 15:21:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:21:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:21:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:21:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:21:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:21:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.37s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 15:21:39 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:21:39 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 15:21:39 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:21:39 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/17 15:21:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:21:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:21:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:21:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:21:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:21:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:21:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:21:58 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:21:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/17 15:22:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:22:09 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 15:22:14 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 15:22:19 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 15:22:19 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:22:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:22:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:22:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:22:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:22:39 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:22:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:22:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:23:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:23:15 Rebalance progress: 12.5
2021/12/17 15:23:20 Rebalance progress: 12.5
2021/12/17 15:23:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:23:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:23:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:23:31 Rebalance progress: 12.5
2021/12/17 15:23:36 Rebalance progress: 12.5
2021/12/17 15:23:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:23:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:23:46 Rebalance progress: 12.5
2021/12/17 15:23:51 Rebalance progress: 12.5
2021/12/17 15:23:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:23:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:24:01 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:24:06 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:24:11 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:24:16 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:24:16 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 15:24:16 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:24:16 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/17 15:24:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:24:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:24:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11201
2021/12/17 15:24:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:24:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:24:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:24:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.53s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 15:24:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:24:44 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 15:24:45 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:24:45 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/17 15:24:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:24:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:24:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:24:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:24:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:24:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:25:03 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:25:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/17 15:25:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:25:14 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 15:25:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 15:25:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/17 15:25:24 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:25:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:25:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:25:45 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:25:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:25:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:25:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:26:16 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:26:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:26:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:26:46 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:26:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:26:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:26:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:27:17 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:27:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:27:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:27:48 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:27:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:27:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:28:04 Rebalance progress: 12.5
2021/12/17 15:28:09 Rebalance progress: 12.5
2021/12/17 15:28:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:28:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:28:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:28:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:28:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:28:19 Rebalance progress: 12.5
2021/12/17 15:28:24 Rebalance progress: 12.5
2021/12/17 15:28:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:28:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:28:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:28:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:28:34 Rebalance progress: 12.5
2021/12/17 15:28:39 Rebalance progress: 12.5
2021/12/17 15:28:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:28:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:28:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:28:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:28:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:28:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:28:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:29:04 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/17 15:29:04 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/17 15:29:04 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:29:04 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/17 15:29:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:29:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:29:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21491
2021/12/17 15:29:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:29:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:29:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:29:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21491
2021/12/17 15:29:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (293.55s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/17 15:29:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 15:29:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/17 15:29:38 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:29:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/17 15:29:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:29:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:29:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:29:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:29:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:29:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:29:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:29:57 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:29:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 15:30:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:30:08 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/17 15:30:13 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/17 15:30:18 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/17 15:30:18 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:30:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:30:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:30:23 Rebalance progress: 12.5
2021/12/17 15:30:28 Rebalance progress: 12.5
2021/12/17 15:30:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:30:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:30:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:30:48 Rebalance progress: 12.5
2021/12/17 15:30:53 Rebalance progress: 12.5
2021/12/17 15:30:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:30:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:30:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:30:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:31:04 Rebalance progress: 12.5
2021/12/17 15:31:09 Rebalance progress: 12.5
2021/12/17 15:31:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:31:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:31:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:31:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:31:19 Rebalance progress: 12.5
2021/12/17 15:31:24 Rebalance progress: 12.5
2021/12/17 15:31:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:31:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:31:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:31:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:31:34 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 15:31:39 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 15:31:44 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 15:31:49 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/17 15:31:49 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/17 15:31:49 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:31:49 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/17 15:31:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:31:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:32:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6911
2021/12/17 15:32:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6911
2021/12/17 15:32:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:32:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:32:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:32:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8592
2021/12/17 15:32:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.53s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/17 15:32:28 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 15:32:28 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/17 15:32:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:32:28 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/17 15:32:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:32:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:32:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:32:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:32:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:32:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:32:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:32:46 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:32:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/17 15:32:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:32:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/17 15:33:02 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/17 15:33:07 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/17 15:33:07 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:33:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/17 15:33:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:33:13 Rebalance progress: 12.5
2021/12/17 15:33:18 Rebalance progress: 12.5
2021/12/17 15:33:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:33:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/17 15:33:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:33:28 Rebalance progress: 12.5
2021/12/17 15:33:33 Rebalance progress: 12.5
2021/12/17 15:33:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:33:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:33:43 Rebalance progress: 12.5
2021/12/17 15:33:48 Rebalance progress: 12.5
2021/12/17 15:33:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:33:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:33:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 15:34:04 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 15:34:09 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 15:34:14 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/17 15:34:14 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/17 15:34:14 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:34:14 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/17 15:34:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:34:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:34:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/17 15:34:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:34:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:34:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:34:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/17 15:34:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.57s)
=== RUN   TestEventingRebInWithOps
2021/12/17 15:34:47 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 15:34:47 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/17 15:34:47 eventingSetup.go:46: No eventing-con process running
2021/12/17 15:34:47 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/17 15:34:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:34:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:34:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:34:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:35:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:35:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:35:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:35:06 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:35:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 15:35:12 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 15:35:17 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 15:35:22 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 15:35:27 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/17 15:35:27 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/17 15:35:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:35:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 15:35:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 15:35:32 Rebalance progress: 12.5
2021/12/17 15:35:37 Rebalance progress: 12.5
2021/12/17 15:35:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:35:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:35:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:35:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:35:47 Rebalance progress: 12.5
2021/12/17 15:35:52 Rebalance progress: 12.5
2021/12/17 15:35:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:35:57 Waiting for src bucket item count to get to: 200000 curr count: 33941
2021/12/17 15:36:02 Waiting for src bucket item count to get to: 200000 curr count: 49850
2021/12/17 15:36:07 Waiting for src bucket item count to get to: 200000 curr count: 49850
2021/12/17 15:36:12 Waiting for src bucket item count to get to: 200000 curr count: 65733
2021/12/17 15:36:17 Waiting for src bucket item count to get to: 200000 curr count: 65733
2021/12/17 15:36:22 Waiting for src bucket item count to get to: 200000 curr count: 80743
2021/12/17 15:36:27 Waiting for src bucket item count to get to: 200000 curr count: 80743
2021/12/17 15:36:32 Waiting for src bucket item count to get to: 200000 curr count: 97049
2021/12/17 15:36:37 Waiting for src bucket item count to get to: 200000 curr count: 97049
2021/12/17 15:36:42 Waiting for src bucket item count to get to: 200000 curr count: 97049
2021/12/17 15:36:47 Waiting for src bucket item count to get to: 200000 curr count: 113182
2021/12/17 15:36:52 Waiting for src bucket item count to get to: 200000 curr count: 113182
2021/12/17 15:36:57 Waiting for src bucket item count to get to: 200000 curr count: 129499
2021/12/17 15:37:02 Waiting for src bucket item count to get to: 200000 curr count: 129499
2021/12/17 15:37:07 Waiting for src bucket item count to get to: 200000 curr count: 145823
2021/12/17 15:37:12 Waiting for src bucket item count to get to: 200000 curr count: 145823
2021/12/17 15:37:17 Waiting for src bucket item count to get to: 200000 curr count: 161724
2021/12/17 15:37:22 Waiting for src bucket item count to get to: 200000 curr count: 161724
2021/12/17 15:37:27 Waiting for src bucket item count to get to: 200000 curr count: 176515
2021/12/17 15:37:33 Waiting for src bucket item count to get to: 200000 curr count: 176515
2021/12/17 15:37:38 Waiting for src bucket item count to get to: 200000 curr count: 192595
2021/12/17 15:37:43 Waiting for src bucket item count to get to: 200000 curr count: 192595
2021/12/17 15:37:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 15:37:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 15:37:48 Starting bucket ops to source bucket
2021/12/17 15:41:38 Waiting for dst bucket item count to get to: 0 curr count: 6196
2021/12/17 15:41:43 Waiting for dst bucket item count to get to: 0 curr count: 6196
2021/12/17 15:41:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:41:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 15:41:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:41:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 15:41:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 15:41:58 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 15:42:03 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/17 15:42:03 App: TestEventingRebInWithOps got un-deployed
2021/12/17 15:42:03 No eventing-con process running
2021/12/17 15:42:03 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/17 15:42:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:42:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:42:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:42:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:42:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:42:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.28s)
=== RUN   TestEventingRebOutWithOps
2021/12/17 15:42:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:42:22 Removing node(s): [] from the cluster
2021/12/17 15:42:27 Rebalance progress: 100
2021/12/17 15:42:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 15:42:32 App: TestEventingRebOutWithOps got un-deployed
2021/12/17 15:42:32 No eventing-con process running
2021/12/17 15:42:32 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/17 15:42:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:42:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:42:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:42:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:42:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:42:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:42:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:42:50 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:42:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 15:42:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 15:43:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 15:43:07 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 15:43:12 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/17 15:43:12 App: TestEventingRebOutWithOps got deployed
2021/12/17 15:43:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:43:12 Starting bucket ops to source bucket
2021/12/17 15:43:17 Rebalance progress: 12.5
2021/12/17 15:43:22 Rebalance progress: 12.5
2021/12/17 15:43:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:43:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:43:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:43:27 Waiting for src bucket item count to get to: 200000 curr count: 11347
2021/12/17 15:43:32 Waiting for src bucket item count to get to: 200000 curr count: 27251
2021/12/17 15:43:37 Waiting for src bucket item count to get to: 200000 curr count: 27251
2021/12/17 15:43:42 Waiting for src bucket item count to get to: 200000 curr count: 43357
2021/12/17 15:43:47 Waiting for src bucket item count to get to: 200000 curr count: 43357
2021/12/17 15:43:52 Waiting for src bucket item count to get to: 200000 curr count: 59598
2021/12/17 15:43:57 Waiting for src bucket item count to get to: 200000 curr count: 59598
2021/12/17 15:44:02 Waiting for src bucket item count to get to: 200000 curr count: 59598
2021/12/17 15:44:07 Waiting for src bucket item count to get to: 200000 curr count: 74334
2021/12/17 15:44:12 Waiting for src bucket item count to get to: 200000 curr count: 74334
2021/12/17 15:44:17 Waiting for src bucket item count to get to: 200000 curr count: 90031
2021/12/17 15:44:22 Waiting for src bucket item count to get to: 200000 curr count: 90031
2021/12/17 15:44:27 Waiting for src bucket item count to get to: 200000 curr count: 105671
2021/12/17 15:44:32 Waiting for src bucket item count to get to: 200000 curr count: 105671
2021/12/17 15:44:37 Waiting for src bucket item count to get to: 200000 curr count: 121647
2021/12/17 15:44:42 Waiting for src bucket item count to get to: 200000 curr count: 121647
2021/12/17 15:44:47 Waiting for src bucket item count to get to: 200000 curr count: 137827
2021/12/17 15:44:52 Waiting for src bucket item count to get to: 200000 curr count: 137827
2021/12/17 15:44:57 Waiting for src bucket item count to get to: 200000 curr count: 154074
2021/12/17 15:45:02 Waiting for src bucket item count to get to: 200000 curr count: 154074
2021/12/17 15:45:07 Waiting for src bucket item count to get to: 200000 curr count: 168909
2021/12/17 15:45:12 Waiting for src bucket item count to get to: 200000 curr count: 168909
2021/12/17 15:45:17 Waiting for src bucket item count to get to: 200000 curr count: 184657
2021/12/17 15:45:22 Waiting for src bucket item count to get to: 200000 curr count: 184657
2021/12/17 15:45:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 15:45:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 15:45:27 Starting bucket ops to source bucket
2021/12/17 15:49:17 Waiting for dst bucket item count to get to: 0 curr count: 5466
2021/12/17 15:49:22 Waiting for dst bucket item count to get to: 0 curr count: 5466
2021/12/17 15:49:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:49:22 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 15:49:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:49:27 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 15:49:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 15:49:38 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 15:49:43 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/17 15:49:43 App: TestEventingRebOutWithOps got un-deployed
2021/12/17 15:49:43 No eventing-con process running
2021/12/17 15:49:43 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/17 15:49:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:49:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:49:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:49:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:49:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (459.68s)
=== RUN   TestEventingSwapRebWithOps
2021/12/17 15:50:06 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 15:50:06 App: TestEventingSwapRebWithOps got un-deployed
2021/12/17 15:50:06 No eventing-con process running
2021/12/17 15:50:06 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/17 15:50:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:50:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:50:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:50:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:50:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:50:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:50:25 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:50:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 15:50:31 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 15:50:36 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 15:50:41 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 15:50:46 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/17 15:50:46 App: TestEventingSwapRebWithOps got deployed
2021/12/17 15:50:46 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 15:50:46 Starting bucket ops to source bucket
2021/12/17 15:50:47 Removing node(s): [] from the cluster
2021/12/17 15:50:52 Rebalance progress: 12.5
2021/12/17 15:50:57 Rebalance progress: 12.5
2021/12/17 15:51:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:51:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:51:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:51:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 15:51:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:51:07 Rebalance progress: 12.5
2021/12/17 15:51:12 Rebalance progress: 12.5
2021/12/17 15:51:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:51:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:51:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:51:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 15:51:22 Rebalance progress: 12.5
2021/12/17 15:51:27 Rebalance progress: 12.5
2021/12/17 15:51:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:51:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:51:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:51:32 Waiting for src bucket item count to get to: 200000 curr count: 66620
2021/12/17 15:51:37 Waiting for src bucket item count to get to: 200000 curr count: 66620
2021/12/17 15:51:42 Waiting for src bucket item count to get to: 200000 curr count: 80990
2021/12/17 15:51:47 Waiting for src bucket item count to get to: 200000 curr count: 80990
2021/12/17 15:51:52 Waiting for src bucket item count to get to: 200000 curr count: 97258
2021/12/17 15:51:57 Waiting for src bucket item count to get to: 200000 curr count: 97258
2021/12/17 15:52:02 Waiting for src bucket item count to get to: 200000 curr count: 113467
2021/12/17 15:52:07 Waiting for src bucket item count to get to: 200000 curr count: 113467
2021/12/17 15:52:12 Waiting for src bucket item count to get to: 200000 curr count: 129550
2021/12/17 15:52:17 Waiting for src bucket item count to get to: 200000 curr count: 129550
2021/12/17 15:52:22 Waiting for src bucket item count to get to: 200000 curr count: 145394
2021/12/17 15:52:27 Waiting for src bucket item count to get to: 200000 curr count: 145394
2021/12/17 15:52:32 Waiting for src bucket item count to get to: 200000 curr count: 161361
2021/12/17 15:52:37 Waiting for src bucket item count to get to: 200000 curr count: 161361
2021/12/17 15:52:42 Waiting for src bucket item count to get to: 200000 curr count: 176327
2021/12/17 15:52:47 Waiting for src bucket item count to get to: 200000 curr count: 176327
2021/12/17 15:52:52 Waiting for src bucket item count to get to: 200000 curr count: 192405
2021/12/17 15:52:57 Waiting for src bucket item count to get to: 200000 curr count: 192405
2021/12/17 15:53:02 Waiting for src bucket item count to get to: 200000 curr count: 192405
2021/12/17 15:53:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 15:53:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 15:53:07 Starting bucket ops to source bucket
2021/12/17 15:56:58 Waiting for dst bucket item count to get to: 0 curr count: 6224
2021/12/17 15:57:03 Waiting for dst bucket item count to get to: 0 curr count: 6224
2021/12/17 15:57:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:57:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 15:57:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 15:57:08 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 15:57:13 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 15:57:18 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 15:57:23 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/17 15:57:23 App: TestEventingSwapRebWithOps got un-deployed
2021/12/17 15:57:23 No eventing-con process running
2021/12/17 15:57:23 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/17 15:57:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:57:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:57:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:57:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:57:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:57:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (460.32s)
=== RUN   TestKVRebInWithOps
2021/12/17 15:57:46 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 15:57:46 App: TestKVRebInWithOps got un-deployed
2021/12/17 15:57:46 No eventing-con process running
2021/12/17 15:57:46 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/17 15:57:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:57:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:57:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 15:57:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:58:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:58:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:58:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 15:58:05 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 15:58:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 15:58:11 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 15:58:16 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 15:58:21 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 15:58:26 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/17 15:58:26 App: TestKVRebInWithOps got deployed
2021/12/17 15:58:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 15:58:26 Starting bucket ops to source bucket
2021/12/17 15:58:27 Removing node(s): [] from the cluster
2021/12/17 15:58:32 Rebalance progress: 12.5
2021/12/17 15:58:37 Rebalance progress: 12.5
2021/12/17 15:58:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:58:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:58:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:58:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 15:58:47 Rebalance progress: 12.5
2021/12/17 15:58:52 Rebalance progress: 12.5
2021/12/17 15:58:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:58:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 15:58:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 15:58:57 Waiting for src bucket item count to get to: 200000 curr count: 34819
2021/12/17 15:59:02 Waiting for src bucket item count to get to: 200000 curr count: 50785
2021/12/17 15:59:07 Waiting for src bucket item count to get to: 200000 curr count: 50785
2021/12/17 15:59:12 Waiting for src bucket item count to get to: 200000 curr count: 67066
2021/12/17 15:59:17 Waiting for src bucket item count to get to: 200000 curr count: 67066
2021/12/17 15:59:22 Waiting for src bucket item count to get to: 200000 curr count: 82055
2021/12/17 15:59:27 Waiting for src bucket item count to get to: 200000 curr count: 82055
2021/12/17 15:59:32 Waiting for src bucket item count to get to: 200000 curr count: 98109
2021/12/17 15:59:37 Waiting for src bucket item count to get to: 200000 curr count: 98109
2021/12/17 15:59:42 Waiting for src bucket item count to get to: 200000 curr count: 114359
2021/12/17 15:59:47 Waiting for src bucket item count to get to: 200000 curr count: 114359
2021/12/17 15:59:52 Waiting for src bucket item count to get to: 200000 curr count: 114359
2021/12/17 15:59:57 Waiting for src bucket item count to get to: 200000 curr count: 130219
2021/12/17 16:00:02 Waiting for src bucket item count to get to: 200000 curr count: 130219
2021/12/17 16:00:07 Waiting for src bucket item count to get to: 200000 curr count: 146314
2021/12/17 16:00:12 Waiting for src bucket item count to get to: 200000 curr count: 146314
2021/12/17 16:00:17 Waiting for src bucket item count to get to: 200000 curr count: 162528
2021/12/17 16:00:22 Waiting for src bucket item count to get to: 200000 curr count: 162528
2021/12/17 16:00:27 Waiting for src bucket item count to get to: 200000 curr count: 177383
2021/12/17 16:00:32 Waiting for src bucket item count to get to: 200000 curr count: 177383
2021/12/17 16:00:37 Waiting for src bucket item count to get to: 200000 curr count: 193487
2021/12/17 16:00:42 Waiting for src bucket item count to get to: 200000 curr count: 193487
2021/12/17 16:00:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 16:00:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 16:00:47 Starting bucket ops to source bucket
2021/12/17 16:04:37 Waiting for dst bucket item count to get to: 0 curr count: 5180
2021/12/17 16:04:42 Waiting for dst bucket item count to get to: 0 curr count: 5180
2021/12/17 16:04:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:04:42 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:04:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:04:47 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 16:04:52 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 16:04:57 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 16:05:02 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/17 16:05:02 App: TestKVRebInWithOps got un-deployed
2021/12/17 16:05:02 No eventing-con process running
2021/12/17 16:05:02 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/17 16:05:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:05:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:05:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:05:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.52s)
=== RUN   TestKVRebOutWithOps
2021/12/17 16:05:21 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:05:21 Removing node(s): [] from the cluster
2021/12/17 16:05:26 Rebalance progress: 100
2021/12/17 16:05:31 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 16:05:31 App: TestKVRebOutWithOps got un-deployed
2021/12/17 16:05:31 No eventing-con process running
2021/12/17 16:05:31 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/17 16:05:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:05:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:05:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:05:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:05:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:05:50 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:05:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 16:05:56 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 16:06:01 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 16:06:06 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 16:06:11 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/17 16:06:11 App: TestKVRebOutWithOps got deployed
2021/12/17 16:06:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:06:11 Starting bucket ops to source bucket
2021/12/17 16:06:16 Rebalance progress: 12.5
2021/12/17 16:06:21 Rebalance progress: 12.5
2021/12/17 16:06:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:06:26 Waiting for src bucket item count to get to: 200000 curr count: 12326
2021/12/17 16:06:31 Waiting for src bucket item count to get to: 200000 curr count: 28144
2021/12/17 16:06:36 Waiting for src bucket item count to get to: 200000 curr count: 28144
2021/12/17 16:06:41 Waiting for src bucket item count to get to: 200000 curr count: 44240
2021/12/17 16:06:46 Waiting for src bucket item count to get to: 200000 curr count: 44240
2021/12/17 16:06:51 Waiting for src bucket item count to get to: 200000 curr count: 60449
2021/12/17 16:06:56 Waiting for src bucket item count to get to: 200000 curr count: 60449
2021/12/17 16:07:01 Waiting for src bucket item count to get to: 200000 curr count: 60449
2021/12/17 16:07:06 Waiting for src bucket item count to get to: 200000 curr count: 75591
2021/12/17 16:07:11 Waiting for src bucket item count to get to: 200000 curr count: 75591
2021/12/17 16:07:16 Waiting for src bucket item count to get to: 200000 curr count: 91462
2021/12/17 16:07:21 Waiting for src bucket item count to get to: 200000 curr count: 91462
2021/12/17 16:07:26 Waiting for src bucket item count to get to: 200000 curr count: 107397
2021/12/17 16:07:31 Waiting for src bucket item count to get to: 200000 curr count: 107397
2021/12/17 16:07:36 Waiting for src bucket item count to get to: 200000 curr count: 123149
2021/12/17 16:07:41 Waiting for src bucket item count to get to: 200000 curr count: 123149
2021/12/17 16:07:46 Waiting for src bucket item count to get to: 200000 curr count: 139285
2021/12/17 16:07:51 Waiting for src bucket item count to get to: 200000 curr count: 139285
2021/12/17 16:07:56 Waiting for src bucket item count to get to: 200000 curr count: 155513
2021/12/17 16:08:01 Waiting for src bucket item count to get to: 200000 curr count: 155513
2021/12/17 16:08:06 Waiting for src bucket item count to get to: 200000 curr count: 170389
2021/12/17 16:08:11 Waiting for src bucket item count to get to: 200000 curr count: 170389
2021/12/17 16:08:16 Waiting for src bucket item count to get to: 200000 curr count: 186239
2021/12/17 16:08:21 Waiting for src bucket item count to get to: 200000 curr count: 186239
2021/12/17 16:08:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 16:08:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 16:08:26 Starting bucket ops to source bucket
2021/12/17 16:12:17 Waiting for dst bucket item count to get to: 0 curr count: 4162
2021/12/17 16:12:22 Waiting for dst bucket item count to get to: 0 curr count: 4162
2021/12/17 16:12:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:12:22 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:12:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:12:27 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 16:12:32 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 16:12:37 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 16:12:42 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/17 16:12:42 App: TestKVRebOutWithOps got un-deployed
2021/12/17 16:12:42 No eventing-con process running
2021/12/17 16:12:42 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/17 16:12:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:12:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:12:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:12:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (459.41s)
=== RUN   TestKVSwapRebWithOps
2021/12/17 16:13:05 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 16:13:05 App: TestKVSwapRebWithOps got un-deployed
2021/12/17 16:13:05 No eventing-con process running
2021/12/17 16:13:05 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/17 16:13:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:13:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:13:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:13:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:13:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:13:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:13:24 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:13:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/17 16:13:30 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 16:13:35 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 16:13:40 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 16:13:45 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/17 16:13:45 App: TestKVSwapRebWithOps got deployed
2021/12/17 16:13:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:13:45 Starting bucket ops to source bucket
2021/12/17 16:13:45 Removing node(s): [] from the cluster
2021/12/17 16:13:50 Rebalance progress: 12.5
2021/12/17 16:13:55 Rebalance progress: 12.5
2021/12/17 16:14:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:14:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:14:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:14:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 16:14:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:14:06 Rebalance progress: 12.5
2021/12/17 16:14:11 Rebalance progress: 12.5
2021/12/17 16:14:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:14:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:14:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:14:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 16:14:21 Rebalance progress: 12.5
2021/12/17 16:14:26 Rebalance progress: 12.5
2021/12/17 16:14:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:14:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:14:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:14:31 Waiting for src bucket item count to get to: 200000 curr count: 67480
2021/12/17 16:14:36 Waiting for src bucket item count to get to: 200000 curr count: 67480
2021/12/17 16:14:41 Waiting for src bucket item count to get to: 200000 curr count: 82085
2021/12/17 16:14:46 Waiting for src bucket item count to get to: 200000 curr count: 82085
2021/12/17 16:14:51 Waiting for src bucket item count to get to: 200000 curr count: 98182
2021/12/17 16:14:56 Waiting for src bucket item count to get to: 200000 curr count: 98182
2021/12/17 16:15:01 Waiting for src bucket item count to get to: 200000 curr count: 114395
2021/12/17 16:15:06 Waiting for src bucket item count to get to: 200000 curr count: 114395
2021/12/17 16:15:11 Waiting for src bucket item count to get to: 200000 curr count: 130197
2021/12/17 16:15:16 Waiting for src bucket item count to get to: 200000 curr count: 130197
2021/12/17 16:15:21 Waiting for src bucket item count to get to: 200000 curr count: 130197
2021/12/17 16:15:26 Waiting for src bucket item count to get to: 200000 curr count: 146177
2021/12/17 16:15:31 Waiting for src bucket item count to get to: 200000 curr count: 146177
2021/12/17 16:15:36 Waiting for src bucket item count to get to: 200000 curr count: 162064
2021/12/17 16:15:41 Waiting for src bucket item count to get to: 200000 curr count: 162064
2021/12/17 16:15:46 Waiting for src bucket item count to get to: 200000 curr count: 176800
2021/12/17 16:15:51 Waiting for src bucket item count to get to: 200000 curr count: 176800
2021/12/17 16:15:56 Waiting for src bucket item count to get to: 200000 curr count: 192977
2021/12/17 16:16:01 Waiting for src bucket item count to get to: 200000 curr count: 192977
2021/12/17 16:16:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/17 16:16:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/17 16:16:06 Starting bucket ops to source bucket
2021/12/17 16:19:56 Waiting for dst bucket item count to get to: 0 curr count: 3903
2021/12/17 16:20:01 Waiting for dst bucket item count to get to: 0 curr count: 3903
2021/12/17 16:20:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:20:01 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:20:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:20:07 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 16:20:12 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 16:20:17 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 16:20:22 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/17 16:20:22 App: TestKVSwapRebWithOps got un-deployed
2021/12/17 16:20:22 No eventing-con process running
2021/12/17 16:20:22 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/17 16:20:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:20:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:20:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:20:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:20:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (459.85s)
=== RUN   TestEventingRebInWithTimers
2021/12/17 16:20:45 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 16:20:45 App: TestEventingRebInWithTimers got un-deployed
2021/12/17 16:20:45 No eventing-con process running
2021/12/17 16:20:45 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/17 16:20:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:20:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:20:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:20:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:20:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:21:04 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:21:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 16:21:10 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 16:21:15 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 16:21:20 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 16:21:25 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/17 16:21:25 App: TestEventingRebInWithTimers got deployed
2021/12/17 16:21:25 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 16:21:25 Starting bucket ops to source bucket
2021/12/17 16:21:26 Removing node(s): [] from the cluster
2021/12/17 16:21:31 Rebalance progress: 12.5
2021/12/17 16:21:36 Rebalance progress: 12.5
2021/12/17 16:21:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:21:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:21:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:21:46 Rebalance progress: 12.5
2021/12/17 16:21:51 Rebalance progress: 12.5
2021/12/17 16:21:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:21:56 Waiting for src bucket item count to get to: 50000 curr count: 28737
2021/12/17 16:22:01 Waiting for src bucket item count to get to: 50000 curr count: 40799
2021/12/17 16:22:06 Waiting for src bucket item count to get to: 50000 curr count: 40799
2021/12/17 16:22:16 Waiting for dst bucket item count to get to: 50000 curr count: 38220
2021/12/17 16:22:21 Waiting for dst bucket item count to get to: 50000 curr count: 38220
2021/12/17 16:22:26 Waiting for dst bucket item count to get to: 50000 curr count: 46232
2021/12/17 16:22:31 Waiting for dst bucket item count to get to: 50000 curr count: 46232
2021/12/17 16:22:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:22:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:22:31 Starting bucket ops to source bucket
2021/12/17 16:23:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:23:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:23:51 Waiting for dst bucket item count to get to: 0 curr count: 49967
2021/12/17 16:23:56 Waiting for dst bucket item count to get to: 0 curr count: 49967
2021/12/17 16:24:01 Waiting for dst bucket item count to get to: 0 curr count: 42029
2021/12/17 16:24:06 Waiting for dst bucket item count to get to: 0 curr count: 42029
2021/12/17 16:24:11 Waiting for dst bucket item count to get to: 0 curr count: 25271
2021/12/17 16:24:16 Waiting for dst bucket item count to get to: 0 curr count: 25271
2021/12/17 16:24:21 Waiting for dst bucket item count to get to: 0 curr count: 4695
2021/12/17 16:24:26 Waiting for dst bucket item count to get to: 0 curr count: 4695
2021/12/17 16:24:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:24:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:24:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:24:31 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 16:24:36 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 16:24:41 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 16:24:46 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/17 16:24:47 App: TestEventingRebInWithTimers got un-deployed
2021/12/17 16:24:47 No eventing-con process running
2021/12/17 16:24:47 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/17 16:24:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:24:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:24:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:25:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (264.94s)
=== RUN   TestEventingRebOutWithTimers
2021/12/17 16:25:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 16:25:06 Removing node(s): [] from the cluster
2021/12/17 16:25:11 Rebalance progress: 100
2021/12/17 16:25:16 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 16:25:16 App: TestEventingRebOutWithTimers got un-deployed
2021/12/17 16:25:16 No eventing-con process running
2021/12/17 16:25:16 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/17 16:25:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:25:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:25:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:25:35 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:25:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 16:25:41 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 16:25:46 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 16:25:51 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 16:25:56 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/17 16:25:56 App: TestEventingRebOutWithTimers got deployed
2021/12/17 16:25:56 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:25:56 Starting bucket ops to source bucket
2021/12/17 16:26:01 Rebalance progress: 12.5
2021/12/17 16:26:06 Rebalance progress: 12.5
2021/12/17 16:26:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:26:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:26:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:26:11 Waiting for src bucket item count to get to: 50000 curr count: 18463
2021/12/17 16:26:16 Waiting for src bucket item count to get to: 50000 curr count: 18463
2021/12/17 16:26:21 Waiting for src bucket item count to get to: 50000 curr count: 27612
2021/12/17 16:26:26 Waiting for src bucket item count to get to: 50000 curr count: 27612
2021/12/17 16:26:31 Waiting for src bucket item count to get to: 50000 curr count: 39444
2021/12/17 16:26:36 Waiting for src bucket item count to get to: 50000 curr count: 39444
2021/12/17 16:26:41 Waiting for src bucket item count to get to: 50000 curr count: 39444
2021/12/17 16:26:46 Waiting for src bucket item count to get to: 50000 curr count: 49908
2021/12/17 16:26:51 Waiting for src bucket item count to get to: 50000 curr count: 49908
2021/12/17 16:27:01 Waiting for dst bucket item count to get to: 50000 curr count: 48420
2021/12/17 16:27:06 Waiting for dst bucket item count to get to: 50000 curr count: 48420
2021/12/17 16:27:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:27:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:27:06 Starting bucket ops to source bucket
2021/12/17 16:28:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:28:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:28:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:28:31 Waiting for dst bucket item count to get to: 0 curr count: 43343
2021/12/17 16:28:36 Waiting for dst bucket item count to get to: 0 curr count: 43343
2021/12/17 16:28:41 Waiting for dst bucket item count to get to: 0 curr count: 31000
2021/12/17 16:28:46 Waiting for dst bucket item count to get to: 0 curr count: 31000
2021/12/17 16:28:51 Waiting for dst bucket item count to get to: 0 curr count: 8316
2021/12/17 16:28:56 Waiting for dst bucket item count to get to: 0 curr count: 8316
2021/12/17 16:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:28:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:28:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:29:02 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 16:29:07 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 16:29:12 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 16:29:17 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/17 16:29:17 App: TestEventingRebOutWithTimers got un-deployed
2021/12/17 16:29:17 No eventing-con process running
2021/12/17 16:29:17 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/17 16:29:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:29:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:29:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:29:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:29:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:29:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (270.03s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/17 16:29:40 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 16:29:40 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/17 16:29:40 No eventing-con process running
2021/12/17 16:29:40 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/17 16:29:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:29:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:29:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:29:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:29:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:29:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:29:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:29:59 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:30:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 16:30:05 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 16:30:10 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 16:30:15 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 16:30:20 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/17 16:30:20 App: TestEventingSwapRebWithTimers got deployed
2021/12/17 16:30:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 16:30:20 Starting bucket ops to source bucket
2021/12/17 16:30:20 Removing node(s): [] from the cluster
2021/12/17 16:30:25 Rebalance progress: 12.5
2021/12/17 16:30:30 Rebalance progress: 12.5
2021/12/17 16:30:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:30:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:30:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:30:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 16:30:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:30:41 Rebalance progress: 12.5
2021/12/17 16:30:46 Rebalance progress: 12.5
2021/12/17 16:30:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:30:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:30:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:30:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 16:30:56 Rebalance progress: 12.5
2021/12/17 16:31:01 Rebalance progress: 12.5
2021/12/17 16:31:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:31:06 Waiting for src bucket item count to get to: 50000 curr count: 46281
2021/12/17 16:31:16 Waiting for dst bucket item count to get to: 50000 curr count: 38824
2021/12/17 16:31:21 Waiting for dst bucket item count to get to: 50000 curr count: 38824
2021/12/17 16:31:26 Waiting for dst bucket item count to get to: 50000 curr count: 49928
2021/12/17 16:31:31 Waiting for dst bucket item count to get to: 50000 curr count: 49928
2021/12/17 16:31:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:31:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:31:31 Starting bucket ops to source bucket
2021/12/17 16:32:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:32:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:32:51 Waiting for dst bucket item count to get to: 0 curr count: 49490
2021/12/17 16:32:56 Waiting for dst bucket item count to get to: 0 curr count: 49490
2021/12/17 16:33:01 Waiting for dst bucket item count to get to: 0 curr count: 38905
2021/12/17 16:33:06 Waiting for dst bucket item count to get to: 0 curr count: 38905
2021/12/17 16:33:11 Waiting for dst bucket item count to get to: 0 curr count: 28923
2021/12/17 16:33:16 Waiting for dst bucket item count to get to: 0 curr count: 28923
2021/12/17 16:33:21 Waiting for dst bucket item count to get to: 0 curr count: 9247
2021/12/17 16:33:26 Waiting for dst bucket item count to get to: 0 curr count: 9247
2021/12/17 16:33:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:33:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:33:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:33:31 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 16:33:36 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 16:33:41 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 16:33:46 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/17 16:33:46 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/17 16:33:46 No eventing-con process running
2021/12/17 16:33:46 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/17 16:33:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:33:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:33:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:33:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:34:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (269.73s)
=== RUN   TestKVRebInWithTimers
2021/12/17 16:34:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 16:34:10 App: TestKVRebInWithTimers got un-deployed
2021/12/17 16:34:10 No eventing-con process running
2021/12/17 16:34:10 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/17 16:34:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:34:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:34:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:34:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:34:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:34:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:34:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:34:29 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:34:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 16:34:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 16:34:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 16:34:45 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 16:34:50 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/17 16:34:50 App: TestKVRebInWithTimers got deployed
2021/12/17 16:34:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:34:50 Starting bucket ops to source bucket
2021/12/17 16:34:50 Removing node(s): [] from the cluster
2021/12/17 16:34:55 Rebalance progress: 12.5
2021/12/17 16:35:00 Rebalance progress: 12.5
2021/12/17 16:35:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:35:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:35:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:35:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:35:10 Rebalance progress: 12.5
2021/12/17 16:35:15 Rebalance progress: 12.5
2021/12/17 16:35:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:35:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:35:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:35:20 Waiting for src bucket item count to get to: 50000 curr count: 34518
2021/12/17 16:35:25 Waiting for src bucket item count to get to: 50000 curr count: 34518
2021/12/17 16:35:30 Waiting for src bucket item count to get to: 50000 curr count: 46397
2021/12/17 16:35:35 Waiting for src bucket item count to get to: 50000 curr count: 46397
2021/12/17 16:35:45 Waiting for dst bucket item count to get to: 50000 curr count: 43786
2021/12/17 16:35:50 Waiting for dst bucket item count to get to: 50000 curr count: 43786
2021/12/17 16:35:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:35:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:35:50 Starting bucket ops to source bucket
2021/12/17 16:37:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:37:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:37:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:37:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:37:21 Waiting for dst bucket item count to get to: 0 curr count: 42601
2021/12/17 16:37:26 Waiting for dst bucket item count to get to: 0 curr count: 42601
2021/12/17 16:37:31 Waiting for dst bucket item count to get to: 0 curr count: 20590
2021/12/17 16:37:36 Waiting for dst bucket item count to get to: 0 curr count: 20590
2021/12/17 16:37:41 Waiting for dst bucket item count to get to: 0 curr count: 7106
2021/12/17 16:37:46 Waiting for dst bucket item count to get to: 0 curr count: 7106
2021/12/17 16:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:37:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:37:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 16:37:56 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 16:38:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 16:38:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/17 16:38:06 App: TestKVRebInWithTimers got un-deployed
2021/12/17 16:38:06 No eventing-con process running
2021/12/17 16:38:06 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/17 16:38:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:38:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:38:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:38:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.66s)
=== RUN   TestKVRebOutWithTimers
2021/12/17 16:38:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:38:25 Removing node(s): [] from the cluster
2021/12/17 16:38:30 Rebalance progress: 100
2021/12/17 16:38:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 16:38:35 App: TestKVRebOutWithTimers got un-deployed
2021/12/17 16:38:35 No eventing-con process running
2021/12/17 16:38:35 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/17 16:38:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:38:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:38:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:38:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:38:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:38:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:38:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:38:54 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:38:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 16:39:00 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 16:39:05 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 16:39:10 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 16:39:15 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/17 16:39:15 App: TestKVRebOutWithTimers got deployed
2021/12/17 16:39:15 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:39:15 Starting bucket ops to source bucket
2021/12/17 16:39:20 Rebalance progress: 12.5
2021/12/17 16:39:25 Rebalance progress: 12.5
2021/12/17 16:39:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:39:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:39:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:39:30 Waiting for src bucket item count to get to: 50000 curr count: 5723
2021/12/17 16:39:35 Waiting for src bucket item count to get to: 50000 curr count: 19532
2021/12/17 16:39:40 Waiting for src bucket item count to get to: 50000 curr count: 29763
2021/12/17 16:39:45 Waiting for src bucket item count to get to: 50000 curr count: 29763
2021/12/17 16:39:50 Waiting for src bucket item count to get to: 50000 curr count: 41784
2021/12/17 16:39:55 Waiting for src bucket item count to get to: 50000 curr count: 41784
2021/12/17 16:40:00 Waiting for src bucket item count to get to: 50000 curr count: 41784
2021/12/17 16:40:10 Waiting for dst bucket item count to get to: 50000 curr count: 40855
2021/12/17 16:40:15 Waiting for dst bucket item count to get to: 50000 curr count: 40855
2021/12/17 16:40:20 Waiting for dst bucket item count to get to: 50000 curr count: 48875
2021/12/17 16:40:25 Waiting for dst bucket item count to get to: 50000 curr count: 48875
2021/12/17 16:40:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:40:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:40:25 Starting bucket ops to source bucket
2021/12/17 16:41:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:41:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:41:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:41:51 Waiting for dst bucket item count to get to: 0 curr count: 46436
2021/12/17 16:41:56 Waiting for dst bucket item count to get to: 0 curr count: 46436
2021/12/17 16:42:01 Waiting for dst bucket item count to get to: 0 curr count: 31297
2021/12/17 16:42:06 Waiting for dst bucket item count to get to: 0 curr count: 31297
2021/12/17 16:42:11 Waiting for dst bucket item count to get to: 0 curr count: 10333
2021/12/17 16:42:16 Waiting for dst bucket item count to get to: 0 curr count: 10333
2021/12/17 16:42:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:42:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:42:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:42:21 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 16:42:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 16:42:31 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 16:42:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/17 16:42:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/17 16:42:36 No eventing-con process running
2021/12/17 16:42:36 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/17 16:42:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:42:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:42:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:42:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:42:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.86s)
=== RUN   TestKVSwapRebWithTimers
2021/12/17 16:42:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:42:59 App: TestKVSwapRebWithTimers got un-deployed
2021/12/17 16:42:59 No eventing-con process running
2021/12/17 16:42:59 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/17 16:43:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:43:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:43:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:43:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:43:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:43:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:43:18 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:43:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/17 16:43:24 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 16:43:29 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 16:43:34 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 16:43:39 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/17 16:43:39 App: TestKVSwapRebWithTimers got deployed
2021/12/17 16:43:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:43:39 Starting bucket ops to source bucket
2021/12/17 16:43:40 Removing node(s): [] from the cluster
2021/12/17 16:43:45 Rebalance progress: 12.5
2021/12/17 16:43:50 Rebalance progress: 12.5
2021/12/17 16:43:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:43:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 16:43:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:44:00 Rebalance progress: 12.5
2021/12/17 16:44:05 Rebalance progress: 12.5
2021/12/17 16:44:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:44:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:44:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:44:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 16:44:15 Rebalance progress: 12.5
2021/12/17 16:44:20 Rebalance progress: 12.5
2021/12/17 16:44:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:44:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:44:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:44:25 Waiting for src bucket item count to get to: 50000 curr count: 45809
2021/12/17 16:44:30 Waiting for src bucket item count to get to: 50000 curr count: 45809
2021/12/17 16:44:40 Waiting for dst bucket item count to get to: 50000 curr count: 44812
2021/12/17 16:44:45 Waiting for dst bucket item count to get to: 50000 curr count: 49365
2021/12/17 16:44:50 Waiting for dst bucket item count to get to: 50000 curr count: 49365
2021/12/17 16:44:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:44:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:44:50 Starting bucket ops to source bucket
2021/12/17 16:46:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:46:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:46:11 Waiting for dst bucket item count to get to: 0 curr count: 47551
2021/12/17 16:46:16 Waiting for dst bucket item count to get to: 0 curr count: 47551
2021/12/17 16:46:21 Waiting for dst bucket item count to get to: 0 curr count: 35259
2021/12/17 16:46:26 Waiting for dst bucket item count to get to: 0 curr count: 35259
2021/12/17 16:46:31 Waiting for dst bucket item count to get to: 0 curr count: 24024
2021/12/17 16:46:36 Waiting for dst bucket item count to get to: 0 curr count: 24024
2021/12/17 16:46:41 Waiting for dst bucket item count to get to: 0 curr count: 2338
2021/12/17 16:46:46 Waiting for dst bucket item count to get to: 0 curr count: 2338
2021/12/17 16:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:46:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/17 16:46:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:46:51 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:46:56 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:47:01 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:47:06 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:47:11 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:47:16 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/17 16:47:16 App: TestKVSwapRebWithTimers got un-deployed
2021/12/17 16:47:16 No eventing-con process running
2021/12/17 16:47:16 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/17 16:47:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:47:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:47:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:47:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:47:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:47:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (280.38s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/17 16:47:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:47:40 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/17 16:47:40 No eventing-con process running
2021/12/17 16:47:40 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/17 16:47:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:47:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:47:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:47:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:47:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:47:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:47:58 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:48:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 16:48:05 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 16:48:10 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 16:48:15 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 16:48:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/17 16:48:20 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/17 16:48:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 16:48:20 Starting bucket ops to source bucket
2021/12/17 16:48:20 Removing node(s): [] from the cluster
2021/12/17 16:48:25 Rebalance progress: 12.5
2021/12/17 16:48:30 Rebalance progress: 12.5
2021/12/17 16:48:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:48:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:48:41 Rebalance progress: 12.5
2021/12/17 16:48:46 Rebalance progress: 12.5
2021/12/17 16:48:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:48:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:48:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:48:51 Waiting for src bucket item count to get to: 50000 curr count: 25404
2021/12/17 16:48:56 Waiting for src bucket item count to get to: 50000 curr count: 35801
2021/12/17 16:49:01 Waiting for src bucket item count to get to: 50000 curr count: 45902
2021/12/17 16:49:06 Waiting for src bucket item count to get to: 50000 curr count: 45902
2021/12/17 16:49:16 Waiting for dst bucket item count to get to: 50000 curr count: 39183
2021/12/17 16:49:21 Waiting for dst bucket item count to get to: 50000 curr count: 39183
2021/12/17 16:49:26 Waiting for dst bucket item count to get to: 50000 curr count: 47530
2021/12/17 16:49:31 Waiting for dst bucket item count to get to: 50000 curr count: 47530
2021/12/17 16:49:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:49:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:49:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:49:36 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:49:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:49:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:49:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:49:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:50:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/17 16:50:01 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/17 16:50:01 No eventing-con process running
2021/12/17 16:50:01 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/17 16:50:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:50:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:50:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:50:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:50:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:50:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:50:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (174.78s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/17 16:50:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 16:50:30 Removing node(s): [] from the cluster
2021/12/17 16:50:35 Rebalance progress: 100
2021/12/17 16:50:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:50:40 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/17 16:50:40 No eventing-con process running
2021/12/17 16:50:40 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/17 16:50:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:50:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:50:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:50:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:50:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:50:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:50:59 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:51:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 16:51:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 16:51:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 16:51:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 16:51:20 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/17 16:51:20 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/17 16:51:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:51:20 Starting bucket ops to source bucket
2021/12/17 16:51:25 Rebalance progress: 12.5
2021/12/17 16:51:30 Rebalance progress: 12.5
2021/12/17 16:51:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:51:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:51:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:51:35 Waiting for src bucket item count to get to: 50000 curr count: 12421
2021/12/17 16:51:40 Waiting for src bucket item count to get to: 50000 curr count: 12421
2021/12/17 16:51:45 Waiting for src bucket item count to get to: 50000 curr count: 26487
2021/12/17 16:51:50 Waiting for src bucket item count to get to: 50000 curr count: 38665
2021/12/17 16:51:55 Waiting for src bucket item count to get to: 50000 curr count: 38665
2021/12/17 16:52:00 Waiting for src bucket item count to get to: 50000 curr count: 38665
2021/12/17 16:52:05 Waiting for src bucket item count to get to: 50000 curr count: 48865
2021/12/17 16:52:10 Waiting for src bucket item count to get to: 50000 curr count: 48865
2021/12/17 16:52:20 Waiting for dst bucket item count to get to: 50000 curr count: 34450
2021/12/17 16:52:25 Waiting for dst bucket item count to get to: 50000 curr count: 34450
2021/12/17 16:52:30 Waiting for dst bucket item count to get to: 50000 curr count: 44341
2021/12/17 16:52:35 Waiting for dst bucket item count to get to: 50000 curr count: 44341
2021/12/17 16:52:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:52:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:52:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:52:40 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:52:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:52:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:52:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:53:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:53:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/17 16:53:05 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/17 16:53:05 No eventing-con process running
2021/12/17 16:53:05 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/17 16:53:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:53:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:53:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:53:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:53:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:53:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.37s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/17 16:53:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:53:39 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/17 16:53:39 No eventing-con process running
2021/12/17 16:53:39 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/17 16:53:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:53:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:53:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:53:58 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:53:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 16:54:04 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 16:54:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 16:54:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 16:54:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/17 16:54:19 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/17 16:54:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 16:54:19 Starting bucket ops to source bucket
2021/12/17 16:54:19 Removing node(s): [] from the cluster
2021/12/17 16:54:25 Rebalance progress: 12.5
2021/12/17 16:54:30 Rebalance progress: 12.5
2021/12/17 16:54:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:54:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:54:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:54:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 16:54:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:54:40 Rebalance progress: 12.5
2021/12/17 16:54:45 Rebalance progress: 12.5
2021/12/17 16:54:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:54:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:54:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:54:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 16:54:55 Rebalance progress: 12.5
2021/12/17 16:55:00 Rebalance progress: 12.5
2021/12/17 16:55:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:55:05 Waiting for src bucket item count to get to: 50000 curr count: 46458
2021/12/17 16:55:10 Waiting for src bucket item count to get to: 50000 curr count: 46458
2021/12/17 16:55:20 Waiting for dst bucket item count to get to: 50000 curr count: 37355
2021/12/17 16:55:25 Waiting for dst bucket item count to get to: 50000 curr count: 37355
2021/12/17 16:55:30 Waiting for dst bucket item count to get to: 50000 curr count: 43935
2021/12/17 16:55:35 Waiting for dst bucket item count to get to: 50000 curr count: 43935
2021/12/17 16:55:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:55:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:55:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:55:40 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:55:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:55:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:55:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:56:00 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:56:05 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/17 16:56:05 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/17 16:56:05 No eventing-con process running
2021/12/17 16:56:05 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/17 16:56:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:56:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:56:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:56:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:56:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:56:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.06s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/17 16:56:39 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:56:39 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/17 16:56:39 No eventing-con process running
2021/12/17 16:56:39 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/17 16:56:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:56:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:56:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:56:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:56:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:56:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:56:58 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 16:56:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 16:57:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 16:57:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 16:57:14 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 16:57:19 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/17 16:57:19 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/17 16:57:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:57:19 Starting bucket ops to source bucket
2021/12/17 16:57:20 Removing node(s): [] from the cluster
2021/12/17 16:57:25 Rebalance progress: 12.5
2021/12/17 16:57:30 Rebalance progress: 12.5
2021/12/17 16:57:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:57:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:57:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 16:57:40 Rebalance progress: 12.5
2021/12/17 16:57:45 Rebalance progress: 12.5
2021/12/17 16:57:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:57:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:57:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 16:57:50 Waiting for src bucket item count to get to: 50000 curr count: 25655
2021/12/17 16:57:55 Waiting for src bucket item count to get to: 50000 curr count: 35490
2021/12/17 16:58:00 Waiting for src bucket item count to get to: 50000 curr count: 35490
2021/12/17 16:58:05 Waiting for src bucket item count to get to: 50000 curr count: 46629
2021/12/17 16:58:10 Waiting for src bucket item count to get to: 50000 curr count: 46629
2021/12/17 16:58:20 Waiting for dst bucket item count to get to: 50000 curr count: 39436
2021/12/17 16:58:25 Waiting for dst bucket item count to get to: 50000 curr count: 39436
2021/12/17 16:58:30 Waiting for dst bucket item count to get to: 50000 curr count: 46902
2021/12/17 16:58:35 Waiting for dst bucket item count to get to: 50000 curr count: 46902
2021/12/17 16:58:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 16:58:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 16:58:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 16:58:40 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:58:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:58:50 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:58:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:59:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:59:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/17 16:59:05 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/17 16:59:05 No eventing-con process running
2021/12/17 16:59:05 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/17 16:59:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:59:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:59:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:59:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:59:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:59:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:59:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 16:59:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.78s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/17 16:59:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 16:59:34 Removing node(s): [] from the cluster
2021/12/17 16:59:39 Rebalance progress: 100
2021/12/17 16:59:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 16:59:44 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/17 16:59:44 No eventing-con process running
2021/12/17 16:59:44 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/17 16:59:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:59:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:59:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 16:59:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 16:59:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 16:59:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:00:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:00:03 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:00:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 17:00:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 17:00:14 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 17:00:19 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 17:00:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/17 17:00:24 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/17 17:00:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:00:24 Starting bucket ops to source bucket
2021/12/17 17:00:29 Rebalance progress: 12.5
2021/12/17 17:00:34 Rebalance progress: 12.5
2021/12/17 17:00:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:00:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:00:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:00:39 Waiting for src bucket item count to get to: 50000 curr count: 6255
2021/12/17 17:00:44 Waiting for src bucket item count to get to: 50000 curr count: 19900
2021/12/17 17:00:49 Waiting for src bucket item count to get to: 50000 curr count: 19900
2021/12/17 17:00:54 Waiting for src bucket item count to get to: 50000 curr count: 33487
2021/12/17 17:00:59 Waiting for src bucket item count to get to: 50000 curr count: 33487
2021/12/17 17:01:04 Waiting for src bucket item count to get to: 50000 curr count: 33487
2021/12/17 17:01:09 Waiting for src bucket item count to get to: 50000 curr count: 43999
2021/12/17 17:01:14 Waiting for src bucket item count to get to: 50000 curr count: 43999
2021/12/17 17:01:24 Waiting for dst bucket item count to get to: 50000 curr count: 30915
2021/12/17 17:01:29 Waiting for dst bucket item count to get to: 50000 curr count: 30915
2021/12/17 17:01:34 Waiting for dst bucket item count to get to: 50000 curr count: 36983
2021/12/17 17:01:39 Waiting for dst bucket item count to get to: 50000 curr count: 36983
2021/12/17 17:01:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:01:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:01:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:01:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 17:01:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 17:01:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 17:01:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 17:02:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 17:02:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/17 17:02:09 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/17 17:02:09 No eventing-con process running
2021/12/17 17:02:10 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/17 17:02:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:02:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:02:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:02:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:02:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:02:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:02:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:02:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:02:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.38s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/17 17:02:48 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:02:48 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/17 17:02:48 No eventing-con process running
2021/12/17 17:02:48 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/17 17:02:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:02:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:02:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:02:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:03:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:03:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:03:07 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:03:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 17:03:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 17:03:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 17:03:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 17:03:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/17 17:03:28 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/17 17:03:28 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:03:28 Starting bucket ops to source bucket
2021/12/17 17:03:28 Removing node(s): [] from the cluster
2021/12/17 17:03:33 Rebalance progress: 12.5
2021/12/17 17:03:38 Rebalance progress: 12.5
2021/12/17 17:03:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:03:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:03:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:03:49 Rebalance progress: 12.5
2021/12/17 17:03:54 Rebalance progress: 12.5
2021/12/17 17:03:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:03:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:03:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:03:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:04:04 Rebalance progress: 12.5
2021/12/17 17:04:09 Rebalance progress: 12.5
2021/12/17 17:04:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:04:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:04:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:04:14 Waiting for src bucket item count to get to: 50000 curr count: 48347
2021/12/17 17:04:19 Waiting for src bucket item count to get to: 50000 curr count: 48347
2021/12/17 17:04:29 Waiting for dst bucket item count to get to: 50000 curr count: 35414
2021/12/17 17:04:34 Waiting for dst bucket item count to get to: 50000 curr count: 35414
2021/12/17 17:04:39 Waiting for dst bucket item count to get to: 50000 curr count: 41531
2021/12/17 17:04:44 Waiting for dst bucket item count to get to: 50000 curr count: 41531
2021/12/17 17:04:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:04:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:04:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:04:49 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:04:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:04:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:05:04 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:05:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:05:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/17 17:05:14 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/17 17:05:14 No eventing-con process running
2021/12/17 17:05:14 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/17 17:05:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:05:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:05:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:05:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:05:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:05:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:05:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.67s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/17 17:05:48 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:05:48 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/17 17:05:48 No eventing-con process running
2021/12/17 17:05:48 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/17 17:05:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:05:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:05:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:06:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:06:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:06:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:06:06 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:06:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 17:06:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 17:06:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 17:06:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 17:06:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/17 17:06:28 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/17 17:06:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 17:06:28 Starting bucket ops to source bucket
2021/12/17 17:06:28 Removing node(s): [] from the cluster
2021/12/17 17:06:33 Rebalance progress: 12.5
2021/12/17 17:06:38 Rebalance progress: 12.5
2021/12/17 17:06:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:06:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:06:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:06:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:06:48 Rebalance progress: 12.5
2021/12/17 17:06:53 Rebalance progress: 12.5
2021/12/17 17:06:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:06:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:06:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:06:58 Waiting for src bucket item count to get to: 50000 curr count: 26596
2021/12/17 17:07:03 Waiting for src bucket item count to get to: 50000 curr count: 36320
2021/12/17 17:07:08 Waiting for src bucket item count to get to: 50000 curr count: 36320
2021/12/17 17:07:13 Waiting for src bucket item count to get to: 50000 curr count: 48073
2021/12/17 17:07:18 Waiting for src bucket item count to get to: 50000 curr count: 48073
2021/12/17 17:07:28 Waiting for dst bucket item count to get to: 50000 curr count: 44361
2021/12/17 17:07:33 Waiting for dst bucket item count to get to: 50000 curr count: 44361
2021/12/17 17:07:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:07:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:07:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:07:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:07:43 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:07:48 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:07:53 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:07:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:08:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/17 17:08:03 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/17 17:08:03 No eventing-con process running
2021/12/17 17:08:04 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/17 17:08:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:08:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:08:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:08:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:08:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:08:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:08:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:08:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.24s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/17 17:08:32 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 17:08:32 Removing node(s): [] from the cluster
2021/12/17 17:08:37 Rebalance progress: 100
2021/12/17 17:08:42 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:08:42 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/17 17:08:42 No eventing-con process running
2021/12/17 17:08:42 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/17 17:08:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:08:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:08:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:08:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:08:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:08:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:09:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:09:01 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:09:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 17:09:07 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 17:09:12 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 17:09:17 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 17:09:22 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/17 17:09:22 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/17 17:09:22 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:09:22 Starting bucket ops to source bucket
2021/12/17 17:09:27 Rebalance progress: 12.5
2021/12/17 17:09:32 Rebalance progress: 12.5
2021/12/17 17:09:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:09:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:09:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:09:37 Waiting for src bucket item count to get to: 50000 curr count: 9467
2021/12/17 17:09:42 Waiting for src bucket item count to get to: 50000 curr count: 22019
2021/12/17 17:09:47 Waiting for src bucket item count to get to: 50000 curr count: 22019
2021/12/17 17:09:52 Waiting for src bucket item count to get to: 50000 curr count: 33395
2021/12/17 17:09:57 Waiting for src bucket item count to get to: 50000 curr count: 33395
2021/12/17 17:10:02 Waiting for src bucket item count to get to: 50000 curr count: 33395
2021/12/17 17:10:07 Waiting for src bucket item count to get to: 50000 curr count: 43510
2021/12/17 17:10:12 Waiting for src bucket item count to get to: 50000 curr count: 43510
2021/12/17 17:10:22 Waiting for dst bucket item count to get to: 50000 curr count: 38449
2021/12/17 17:10:27 Waiting for dst bucket item count to get to: 50000 curr count: 38449
2021/12/17 17:10:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:10:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:10:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:10:32 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:10:38 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:10:43 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:10:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:10:53 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:10:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/17 17:10:58 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/17 17:10:58 No eventing-con process running
2021/12/17 17:10:58 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/17 17:11:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:11:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:11:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:11:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:11:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:11:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.32s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/17 17:11:26 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:11:26 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/17 17:11:26 No eventing-con process running
2021/12/17 17:11:26 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/17 17:11:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:11:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:11:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:11:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:11:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:11:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:11:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:11:45 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:11:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 17:11:51 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 17:11:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 17:12:01 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 17:12:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/17 17:12:06 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/17 17:12:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/17 17:12:06 Starting bucket ops to source bucket
2021/12/17 17:12:07 Removing node(s): [] from the cluster
2021/12/17 17:12:12 Rebalance progress: 12.5
2021/12/17 17:12:17 Rebalance progress: 12.5
2021/12/17 17:12:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:12:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:12:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:12:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 17:12:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:12:28 Rebalance progress: 12.5
2021/12/17 17:12:33 Rebalance progress: 12.5
2021/12/17 17:12:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:12:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:12:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:12:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:12:43 Rebalance progress: 12.5
2021/12/17 17:12:48 Rebalance progress: 12.5
2021/12/17 17:12:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:12:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:12:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:12:53 Waiting for src bucket item count to get to: 50000 curr count: 49297
2021/12/17 17:12:58 Waiting for src bucket item count to get to: 50000 curr count: 49297
2021/12/17 17:13:08 Waiting for dst bucket item count to get to: 50000 curr count: 39852
2021/12/17 17:13:13 Waiting for dst bucket item count to get to: 50000 curr count: 39852
2021/12/17 17:13:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:13:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:13:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:13:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:13:23 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:13:28 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:13:33 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:13:38 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:13:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/17 17:13:43 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/17 17:13:43 No eventing-con process running
2021/12/17 17:13:43 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/17 17:13:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:13:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:13:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:13:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:14:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.51s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/17 17:14:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 17:14:17 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/17 17:14:17 No eventing-con process running
2021/12/17 17:14:17 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/17 17:14:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:14:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:14:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:14:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:14:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:14:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:14:36 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:14:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/17 17:14:42 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 17:14:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 17:14:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 17:14:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/17 17:14:57 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/17 17:14:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:14:57 Starting bucket ops to source bucket
2021/12/17 17:14:57 Removing node(s): [] from the cluster
2021/12/17 17:15:02 Rebalance progress: 12.5
2021/12/17 17:15:07 Rebalance progress: 12.5
2021/12/17 17:15:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:15:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:15:17 Rebalance progress: 12.5
2021/12/17 17:15:22 Rebalance progress: 12.5
2021/12/17 17:15:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:15:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:15:27 Waiting for src bucket item count to get to: 50000 curr count: 26348
2021/12/17 17:15:32 Waiting for src bucket item count to get to: 50000 curr count: 36971
2021/12/17 17:15:37 Waiting for src bucket item count to get to: 50000 curr count: 36971
2021/12/17 17:15:42 Waiting for src bucket item count to get to: 50000 curr count: 47994
2021/12/17 17:15:47 Waiting for src bucket item count to get to: 50000 curr count: 47994
2021/12/17 17:15:57 Waiting for dst bucket item count to get to: 50000 curr count: 46395
2021/12/17 17:16:02 Waiting for dst bucket item count to get to: 50000 curr count: 46395
2021/12/17 17:16:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:16:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:16:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:16:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 17:16:12 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 17:16:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 17:16:23 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 17:16:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/17 17:16:28 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/17 17:16:28 No eventing-con process running
2021/12/17 17:16:28 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/17 17:16:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:16:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:16:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:16:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:16:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:16:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (159.30s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/17 17:16:51 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:16:51 Removing node(s): [] from the cluster
2021/12/17 17:16:57 Rebalance progress: 100
2021/12/17 17:17:02 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:17:02 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/17 17:17:02 No eventing-con process running
2021/12/17 17:17:02 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/17 17:17:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:17:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:17:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:17:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:17:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:17:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:17:20 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:17:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/17 17:17:26 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 17:17:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 17:17:36 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 17:17:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/17 17:17:41 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/17 17:17:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:17:41 Starting bucket ops to source bucket
2021/12/17 17:17:46 Rebalance progress: 12.5
2021/12/17 17:17:51 Rebalance progress: 12.5
2021/12/17 17:17:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:17:56 Waiting for src bucket item count to get to: 50000 curr count: 10721
2021/12/17 17:18:01 Waiting for src bucket item count to get to: 50000 curr count: 24253
2021/12/17 17:18:06 Waiting for src bucket item count to get to: 50000 curr count: 24253
2021/12/17 17:18:12 Waiting for src bucket item count to get to: 50000 curr count: 35107
2021/12/17 17:18:17 Waiting for src bucket item count to get to: 50000 curr count: 35107
2021/12/17 17:18:22 Waiting for src bucket item count to get to: 50000 curr count: 45652
2021/12/17 17:18:27 Waiting for src bucket item count to get to: 50000 curr count: 45652
2021/12/17 17:18:32 Waiting for src bucket item count to get to: 50000 curr count: 45652
2021/12/17 17:18:42 Waiting for dst bucket item count to get to: 50000 curr count: 34605
2021/12/17 17:18:47 Waiting for dst bucket item count to get to: 50000 curr count: 34605
2021/12/17 17:18:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:18:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:18:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:18:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:18:57 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:19:02 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:19:07 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:19:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:19:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/17 17:19:17 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/17 17:19:17 No eventing-con process running
2021/12/17 17:19:17 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/17 17:19:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:19:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:19:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:19:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:19:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:19:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:19:41 Waiting for dst bucket item count to get to: 0 curr count: 28887
2021/12/17 17:19:46 Waiting for dst bucket item count to get to: 0 curr count: 28887
2021/12/17 17:19:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.30s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/17 17:19:56 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 17:19:56 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/17 17:19:56 No eventing-con process running
2021/12/17 17:19:56 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/17 17:19:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:20:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:20:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:20:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:20:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:20:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:20:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:20:14 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:20:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/17 17:20:20 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 17:20:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 17:20:30 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 17:20:35 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/17 17:20:35 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/17 17:20:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:20:35 Starting bucket ops to source bucket
2021/12/17 17:20:36 Removing node(s): [] from the cluster
2021/12/17 17:20:41 Rebalance progress: 12.5
2021/12/17 17:20:46 Rebalance progress: 12.5
2021/12/17 17:20:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:20:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:20:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:20:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:20:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:20:56 Rebalance progress: 12.5
2021/12/17 17:21:01 Rebalance progress: 12.5
2021/12/17 17:21:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:21:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:21:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:21:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:21:11 Rebalance progress: 12.5
2021/12/17 17:21:16 Rebalance progress: 12.5
2021/12/17 17:21:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:21:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:21:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:21:21 Waiting for src bucket item count to get to: 50000 curr count: 48445
2021/12/17 17:21:26 Waiting for src bucket item count to get to: 50000 curr count: 48445
2021/12/17 17:21:36 Waiting for dst bucket item count to get to: 50000 curr count: 42212
2021/12/17 17:21:41 Waiting for dst bucket item count to get to: 50000 curr count: 42212
2021/12/17 17:21:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/17 17:21:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/17 17:21:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:21:47 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 17:21:52 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 17:21:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 17:22:02 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 17:22:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/17 17:22:07 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/17 17:22:07 No eventing-con process running
2021/12/17 17:22:07 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/17 17:22:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:22:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:22:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/17 17:22:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:22:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:22:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:22:30 Waiting for dst bucket item count to get to: 0 curr count: 21391
2021/12/17 17:22:35 Waiting for dst bucket item count to get to: 0 curr count: 21391
2021/12/17 17:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.84s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/17 17:22:45 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:22:45 App: bucket_op_on_update got un-deployed
2021/12/17 17:22:45 No eventing-con process running
2021/12/17 17:22:45 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:22:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:22:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:22:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:22:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:22:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:22:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:23:04 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:23:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:23:10 Starting bucket ops to source bucket
2021/12/17 17:23:15 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:23:20 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:23:25 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:23:25 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:23:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:23:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:23:30 Rebalance progress: 12.5
2021/12/17 17:23:35 Rebalance progress: 12.5
2021/12/17 17:23:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:23:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:23:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:23:46 Rebalance progress: 12.5
2021/12/17 17:23:51 Rebalance progress: 12.5
2021/12/17 17:23:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:23:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:23:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 17:23:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:24:01 Rebalance progress: 12.5
2021/12/17 17:24:06 Rebalance progress: 12.5
2021/12/17 17:24:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:24:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:24:16 Rebalance progress: 12.5
2021/12/17 17:24:21 Rebalance progress: 12.5
2021/12/17 17:24:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:24:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:24:31 Rebalance progress: 12.5
2021/12/17 17:24:36 Rebalance progress: 12.5
2021/12/17 17:24:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:24:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 17:24:46 Rebalance progress: 12.5
2021/12/17 17:24:51 Rebalance progress: 12.5
2021/12/17 17:24:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:24:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:24:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:25:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:25:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:25:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:25:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:25:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:25:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:25:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:25:27 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:25:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:25:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:25:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:25:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10616
2021/12/17 17:25:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:25:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:25:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:25:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3915
2021/12/17 17:25:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3915
2021/12/17 17:25:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (199.90s)
=== RUN   TestKVForceFailover
2021/12/17 17:26:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:26:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:26:10 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:26:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:26:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:26:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:26:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:26:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:26:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:26:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:26:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:26:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:26:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:26:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:26:29 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:26:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:26:31 Starting bucket ops to source bucket
2021/12/17 17:26:36 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:26:41 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:26:46 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:26:51 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:26:51 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:26:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:26:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:26:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/17 17:27:11 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/17 17:27:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/17 17:27:16 Rebalance progress: 12.5
2021/12/17 17:27:21 Rebalance progress: 12.5
2021/12/17 17:27:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:27:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:27:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:27:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/17 17:27:31 Rebalance progress: 12.5
2021/12/17 17:27:36 Rebalance progress: 12.5
2021/12/17 17:27:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:27:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:27:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:27:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:27:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:28:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:28:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:28:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:28:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:28:12 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:28:12 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:28:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:28:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:28:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/17 17:28:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7043
2021/12/17 17:28:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:28:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:28:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:28:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2103
2021/12/17 17:28:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2103
2021/12/17 17:28:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (170.01s)
=== RUN   TestKVHardFailoverRecovery
2021/12/17 17:28:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:28:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:28:55 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:28:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:28:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:29:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:29:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:29:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:29:14 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:29:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:29:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:29:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:29:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:29:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:29:35 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:29:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:29:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:29:40 Rebalance progress: 12.5
2021/12/17 17:29:45 Rebalance progress: 12.5
2021/12/17 17:29:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:29:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:29:50 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/17 17:30:50 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/17 17:30:56 Rebalance progress: 12.5
2021/12/17 17:31:01 Rebalance progress: 12.5
2021/12/17 17:31:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:31:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:31:11 Rebalance progress: 12.5
2021/12/17 17:31:16 Rebalance progress: 12.5
2021/12/17 17:31:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:31:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:31:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:31:21 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:31:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:31:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:31:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:31:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:31:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:31:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:31:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:31:51 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:31:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:31:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:31:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:32:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12049
2021/12/17 17:32:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12049
2021/12/17 17:32:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:32:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:32:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:32:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.38s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/17 17:32:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:32:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:32:25 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:32:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:32:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:32:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:32:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:32:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:32:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:32:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:32:43 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:32:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:32:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:32:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:33:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:33:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:33:05 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:33:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:33:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:33:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:33:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:33:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 17:33:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:33:11 Rebalance progress: 12.5
2021/12/17 17:33:16 Rebalance progress: 12.5
2021/12/17 17:33:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:33:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:33:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:33:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/17 17:33:26 Rebalance progress: 12.5
2021/12/17 17:33:31 Rebalance progress: 12.5
2021/12/17 17:33:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:33:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:33:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:33:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:33:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:33:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:33:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:33:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:34:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:34:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:34:01 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:34:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:34:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:34:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:34:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4618
2021/12/17 17:34:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4618
2021/12/17 17:34:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:34:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:34:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:34:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/17 17:34:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 49
2021/12/17 17:34:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.27s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/17 17:34:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:34:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:34:45 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:34:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:34:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:34:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:34:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:34:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:34:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:34:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:34:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:34:59 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:35:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:35:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:35:10 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:35:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:35:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:35:20 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:35:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:35:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:35:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:35:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:35:25 Rebalance progress: 12.5
2021/12/17 17:35:30 Rebalance progress: 12.5
2021/12/17 17:35:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:35:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:35:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:35:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:35:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:35:40 Rebalance progress: 12.5
2021/12/17 17:35:45 Rebalance progress: 12.5
2021/12/17 17:35:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:35:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:35:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:35:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:35:55 Rebalance progress: 12.5
2021/12/17 17:36:00 Rebalance progress: 12.5
2021/12/17 17:36:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:36:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:36:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:36:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:36:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:36:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:36:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:36:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:36:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:36:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:36:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:36:36 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:36:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:36:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:36:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:36:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6060
2021/12/17 17:36:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:36:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:36:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:36:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6060
2021/12/17 17:36:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:37:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:37:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:37:04 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:37:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:37:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:37:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:37:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:37:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:37:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:37:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (163.03s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/17 17:37:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:37:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:37:28 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:37:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:37:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:37:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:37:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:37:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:37:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:37:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:37:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:37:47 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 17:37:47 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:37:47 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/17 17:37:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:37:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:37:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:37:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:38:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:38:00 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:38:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:38:01 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:38:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 17:38:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:38:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:38:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:38:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:38:22 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/17 17:38:27 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 17:38:32 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 17:38:37 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 17:38:37 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:38:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:38:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:38:43 Rebalance progress: 12.5
2021/12/17 17:38:48 Rebalance progress: 12.5
2021/12/17 17:38:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:38:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:38:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:38:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:38:58 Rebalance progress: 12.5
2021/12/17 17:39:03 Rebalance progress: 12.5
2021/12/17 17:39:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:39:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/17 17:39:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:39:13 Rebalance progress: 12.5
2021/12/17 17:39:18 Rebalance progress: 12.5
2021/12/17 17:39:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:39:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:39:29 Rebalance progress: 12.5
2021/12/17 17:39:34 Rebalance progress: 12.5
2021/12/17 17:39:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:39:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:39:44 Rebalance progress: 12.5
2021/12/17 17:39:49 Rebalance progress: 12.5
2021/12/17 17:39:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:39:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:39:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/17 17:39:59 Rebalance progress: 12.5
2021/12/17 17:40:04 Rebalance progress: 12.5
2021/12/17 17:40:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:40:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:40:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:40:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:40:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:40:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:40:44 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:40:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:40:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:40:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:40:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12139
2021/12/17 17:40:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:41:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:41:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:41:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:41:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:41:10 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:41:15 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:41:20 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:41:25 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:41:30 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:41:35 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:41:35 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 17:41:35 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:41:35 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/17 17:41:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:41:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:41:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:41:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:41:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:41:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (270.52s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/17 17:41:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:41:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:41:59 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:41:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:42:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:42:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:42:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:42:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:42:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:42:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:42:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:42:17 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 17:42:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:42:17 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/17 17:42:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:42:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:42:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:42:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:42:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:42:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:42:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:42:31 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:42:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:42:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:42:32 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:42:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/17 17:42:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:42:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:42:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:42:53 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:42:53 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/17 17:42:58 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 17:43:03 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 17:43:08 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/17 17:43:08 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:43:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:43:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:43:13 Rebalance progress: 12.5
2021/12/17 17:43:18 Rebalance progress: 12.5
2021/12/17 17:43:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:43:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:43:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:43:29 Rebalance progress: 12.5
2021/12/17 17:43:34 Rebalance progress: 12.5
2021/12/17 17:43:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:43:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:43:44 Rebalance progress: 12.5
2021/12/17 17:43:49 Rebalance progress: 12.5
2021/12/17 17:43:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:43:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:43:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:43:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:44:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:44:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:44:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:44:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:44:20 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:44:20 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:44:20 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:44:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:44:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:44:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7627
2021/12/17 17:44:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7627
2021/12/17 17:44:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:44:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:44:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:44:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6346
2021/12/17 17:44:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6346
2021/12/17 17:44:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:44:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:45:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:45:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:45:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:45:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:45:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:45:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/17 17:45:26 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/17 17:45:26 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:45:26 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/17 17:45:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:45:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:45:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:45:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:45:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:45:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (231.36s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/17 17:45:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:45:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:45:50 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:45:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:45:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:45:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:45:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:45:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:46:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:46:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:46:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:46:08 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:46:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:46:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:46:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:46:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:46:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:46:30 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:46:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:46:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:46:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:46:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:46:50 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:46:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:46:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:47:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:47:06 Rebalance progress: 12.5
2021/12/17 17:47:11 Rebalance progress: 12.5
2021/12/17 17:47:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:47:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:47:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:47:21 Rebalance progress: 12.5
2021/12/17 17:47:26 Rebalance progress: 12.5
2021/12/17 17:47:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:47:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:47:36 Rebalance progress: 12.5
2021/12/17 17:47:41 Rebalance progress: 12.5
2021/12/17 17:47:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:47:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:47:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:47:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:47:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:48:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:48:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:48:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:48:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:48:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:48:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:48:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:48:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:48:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:48:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9150
2021/12/17 17:48:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:48:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:48:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:48:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3053
2021/12/17 17:48:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3053
2021/12/17 17:48:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.41s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/17 17:48:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:48:55 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:48:55 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:48:55 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/17 17:48:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:49:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:49:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:49:14 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:49:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:49:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/17 17:49:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:49:30 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:49:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:49:35 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:49:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:49:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:49:41 Rebalance progress: 12.5
2021/12/17 17:49:46 Rebalance progress: 12.5
2021/12/17 17:49:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:49:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:49:51 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:50:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:50:06 Rebalance progress: 12.5
2021/12/17 17:50:11 Rebalance progress: 12.5
2021/12/17 17:50:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:50:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/17 17:50:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/17 17:50:22 Rebalance progress: 12.5
2021/12/17 17:50:27 Rebalance progress: 12.5
2021/12/17 17:50:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:50:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/17 17:50:37 Rebalance progress: 12.5
2021/12/17 17:50:42 Rebalance progress: 12.5
2021/12/17 17:50:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:50:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:50:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:50:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:50:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:51:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:51:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:51:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:51:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:51:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:51:17 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:51:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:51:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:51:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:51:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8680
2021/12/17 17:51:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:51:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:51:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:51:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7741
2021/12/17 17:51:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7741
2021/12/17 17:51:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.67s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/17 17:51:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/17 17:51:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/17 17:51:56 Rebalance progress: 100
2021/12/17 17:51:56 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:52:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:52:26 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/17 17:52:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/17 17:52:32 Starting bucket ops to source bucket
2021/12/17 17:52:32 Removing node(s): [] from the cluster
2021/12/17 17:52:37 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/17 17:52:42 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:52:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/17 17:52:48 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:52:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:52:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/17 17:52:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/17 17:52:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:52:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:53:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:53:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:53:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:53:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/17 17:53:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/17 17:53:18 eventingSetup.go:46: No eventing-con process running
2021/12/17 17:53:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/17 17:53:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:53:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:53:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/17 17:53:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/17 17:53:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/17 17:53:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1508
2021/12/17 17:53:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.56s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12042.972s

Rebalance tests took 3 hours, 20 minutes and 45 seconds.

Testing Succeeded


Version: versions-17.12.2021-10.09.cfg
Build Log: make-17.12.2021-10.09.log
Server Log: logs-17.12.2021-10.09.tar.gz

Finished